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Forward:  About  This  Document 


This  document  does  not  detail  all  activities  and  data  associated  with  the  Aircraft 
Maintenance  Intuitive  Troubleshooting  (AMIT)  program;  rather,  it  provides  the  reader  with  an 
overview  of  AMIT  program  activities,  with  detailed  focus  on  the  AMIT  Field  Demonstration 
Test  (FDT)  and  Results.  Referring  to  the  documents  cited  in  the  bibliography  will  provide  the 
reader  with  detailed  descriptions  of  other  relevant  program  activities  and  results. 

In  this  document,  the  AMIT  team  attempts  to  strike  a  balance  in  describing,  in  as  concise 
and  understandable  a  manner  as  possible,  the  execution  and  results  of  the  AMIT  program  to  three 
disparate  groups  of  readers: 

•  the  scientific  community,  whose  primary  interest  is  the  applicability  of  the  research  and 
test  methods  used,  the  validity  of  the  results,  and  the  effort’s  contribution  to  the  body  of 
scientific  knowledge 

•  the  aircraft  maintenance  community,  whose  interests  include  the  outcome  of  the  test  and 
the  improvements  the  solution  offers  in  the  work  place 

•  Air  Force  commanders  and  executives,  who  want  assurance  that  fielding  the  AMIT 
solution  is  useful  and  cost-efficient 


Preface 


The  impetus  for  the  AMIT  program  was  twofold.  First,  troubleshooting-intensive  aircraft 
maintenance  career  fields  have  undergone  over  40  years  of  consolidations,  generalizations,  and 
manpower  cuts,  while  operational  tempo  and  weapon  system  complexity  have  increased.  (For 
example,  the  current  “Avionics”  Air  Force  Specialty  Code  (AFSC)  was  once  seven  separate 
career  fields,  as  illustrated  below.)  Second,  information  technologies  and  applications  have 
matured  to  the  point  they  can  be  leveraged  to  significantly  improve  aircraft  maintenance  and 
troubleshooting  processes. 
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•  1960’s  AFSCs  (Avionics  =  7  career  fields)  -  highly  specialized  training,  not  weapon 
system  specific 

•  1970’s  -  AFSC  mergers  begin  to  become  less  technically  specialized 

•  1980’s  -  more  mergers  even  less  specialized,  increasing  weapon  system  specialization 
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•  1990’s  -  more  mergers  begin  weapon  system  specialization  dilution  (more  than  one 
aircraft) 

•  2000 ’s  -  final  merger  (Avionics  now  one  career  field) 

•  Bottom  line:  Technicians  now  receive  less  training,  are  less  specialized,  and  thus  require 
more  on-the-job-training  at  their  first  assignment.  These  conditions  are  exacerbated  by 
expert  attrition  and  daily  workload  due  to  force  reductions  and  high  operational  tempos. 

The  Aircraft  Maintenance  Intuitive  Troubleshooting  (AMIT)  program  was  conceived  at 
the  U.S.  Air  Force  Research  Laboratory’s  Human  Effectiveness  Directorate  as  a  revolutionary 
step  in  providing  domain-specific  knowledge  and  information  to  those  in  aircraft  troubleshooting 
roles.  AMIT’s  objective  was  to  integrate  modem  approaches  to  human-computer  interaction, 
knowledge-centric  enterprise  data  management,  interactive  electronic  technical  manuals,  and 
collaboration  tools  into  a  synergistic  capability  that  can  improve  human  performance.  The 
resulting  solution  integrates  and  presents  complete  holistic  knowledge  and  information  at  the 
point  of  task,  allowing  technicians  to  more  effectively  troubleshoot  aircraft  system  problems. 
This  capability  was  demonstrated  in  a  field  test  that  objectively  measured  performance,  including 
task  times,  error  rates,  and  user  acceptance. 
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1  Summary 

1.1  Overview 

Aircraft  Maintenance  Intuitive  Troubleshooting  (AMIT)  was  conceived  at  an 
AFRL/MAJCOM  technical  interchange  meeting,  with  the  realization  that  technician  performance 
was  an  under  explored  and  possibly  significant  contributor  to  aircraft  availability  and  readiness. 
Heretofore,  the  Air  Force  (AF)  has  given  much  attention  to  maintenance  levels,  repair  locations, 
spares,  tools  and  test  equipment,  technician  training,  specialty  codes,  and  even  to  job 
performance  rating.  Little  attention,  however,  has  been  paid  to  factors  affecting  the  technician’s 
performance  -  the  “how”  and  “why”  technicians  perform  the  way  they  do  on  the  flightline.  How 
do  they  successfully  return  highly  complex  aircraft  to  service  in  spite  of  the  known  limitations 
and  challenges  imposed  by  the  maintenance  process  and  related  support  and  supply  systems? 

Given  this  line  of  thinking,  the  Logistics  Readiness  Branch  of  the  Human  Effectiveness 
Directorate  of  the  Air  Force  Research  Laboratory  (AFRL/HEAL)  initiated  a  6.3  advanced 
research  and  development  (R&D)  program  and  awarded  the  AMIT  contract  to  NCI  Information 
Systems,  Incorporated  (NCI)  in  December  2003.  NCI  is  an  information  technology  services 
provider  with  relevant  skills  in  the  aircraft  maintenance  and  maintenance  management  domains. 

Discussions  with  subject  matter  experts  (SMEs)  in  the  program’s  formative  days  focused 
attention  on  the  decision  points  in  the  maintenance  process;  specifically  the  decisions  a 
technician  makes  while  isolating  a  failed  component.  When  a  technician  selects  the  correct  fault 
isolation  path  or  opts  to  remove  and  replace  the  appropriate  line  replaceable  unit  (LRU),  time  is 
effectively  used,  resource  consumption  minimized,  and  the  aircraft  is  made  available  to  perform 
its  mission  in  minimum  time. 
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The  SMEs  pointed  out  that  fault  isolation  time  or  troubleshooting  time  is  one  of  many 
contributors  to  an  individual  aircraft’s  total  downtime.  Other  significant  contributors  to  aircraft 
downtime  include  routine  maintenance,  periodic  inspections,  and  repairs  or  replacements  of 
routine  problems  like  a  worn  or  flat  tire.  Estimates  validated  by  information  from  squadron 
preparedness  briefings  and  ad  hoc  reports  from  the  Integrated  Maintenance  Data  System 
(IMDS),  indicated  25-30%  of  all  documented  aircraft  maintenance  activities  involve  some  form 
of  troubleshooting.  Therefore,  a  significant  improvement  in  troubleshooting  efficiency  logically 
ought  to  have  a  material  impact  on  downtime  at  an  aggregated  level  such  as  a  squadron.  While 
any  improvement  ought  to  be  evident  at  the  organizational  level,  troubleshooting  efficiency  can 
only  be  achieved  by  improving  the  average  troubleshooting  proficiency  of  the  organization’s 
technicians. 

After  preliminary  discussions,  the  AMIT  Team  decided  to  spend  program  resources 
researching  ways  to  significantly  improve  Novice  and  Expert  maintainers’  troubleshooting 
proficiency,  thereby  increasing  the  organization’s  total  aircraft  availability.  The  Team  did  not 
attempt  to  determine  how  large  the  average  proficiency  improvement  would  need  to  be  to 
become  significant  or  even  how  much  improvement  was  possible  for  the  average  maintainer.  If 
the  former  approach  were  successful,  the  point  would  be  proved  with  a  developed  and  field- 
demonstrated  solution.  Following  the  later  course  would,  at  best,  simply  prove  such  an 
improvement  were  possible  and  yield  yet  another  metric  for  the  AF  to  collect. 

For  this  effort,  the  AMIT  Team  defined  troubleshooting  as  “a  process  that  includes  all 
activities  related  to  determining  the  correct  repair  action.”  This  included  all  activities  from 
problem  notification  through  repair  verification;  understanding  the  current  discrepancy,  technical 
data  research,  historical  maintenance  data  research,  determining  aircraft  condition,  and 
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communication.  Two  factors  affecting  troubleshooting  efficiency  were  considered:  task  time,  or 
the  time  taken  to  effectively  isolate  a  faulty  component,  and  task  accuracy,  a  measure  of  the 
effectiveness  of  the  decisions  made  during  the  troubleshooting  process.  Every  experienced 
troubleshooter  refers  to  some  documents,  establishes  some  mental  reference  points,  and  then 
takes  a  mental  leap  from  observed  symptoms  to  solution,  trusting  his  or  her  intuition  to  guide  the 
direction  and  distance  of  the  leap.  Hence,  the  word  “intuitive”  was  included  in  the  AMIT 
program  title,  to  emphasize  the  cognitive  dimension  of  troubleshooting. 

By  emphasizing  the  cognitive  aspect  of  troubleshooting,  the  Team  resolved  to  research 
and  develop  an  effective  method(s)  of  helping  troubleshooters  make  well-informed  decisions  by 
providing  complete,  timely,  and  contextually  relevant  data,  information,  and  knowledge  at  the 
point  of  task.  Troubleshooters  who  more  quickly  form  an  accurate  mental  model  will  not  only 
take  less  time  to  troubleshoot  each  failure,  but,  over  time,  their  mental  model  would  expand  in 
size  and  increase  in  accuracy,  resulting  in  each  intuitive  act  becoming  a  well-placed  step  rather 
than  a  blind  leap. 

When  matched  with  the  necessary  technical  tasks  and  the  program’s  37-month  duration, 
AFRL’s  desire  for  a  three-phase  approach  almost  naturally  organized  the  effort  into  three  12- 
month  phases  along  the  following  lines: 

•  Defining  Customer  Needs  and  Processes:  A  period  of  research  and  data  gathering 
activities  including  extensive  literature  reviews,  cognitive  task  analyses  (CTA),  user 
process  interviews,  product  evaluations,  user  groups,  and  technical  interchange  meetings 
along  with  several  iterations  of  analysis,  summarizations,  and  data  reductions.  The  result 
was  a  complete,  current,  and  comprehensive  understanding  of  aircraft  system 
troubleshooting  on  a  typical  AF  flightline.  The  resulting  set  of  functional  needs,  or 
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requirements,  for  the  AMIT  solution  took  into  account  related  maintenance  processes  and 
issues,  as  well  as  technician  attitudes,  thought  patterns,  and  protocols. 

•  Design  and  Development:  Early  in  this  phase,  the  AMIT  solution’s  five  primary  features 
crystallized:  task-specific  electronic  technical  data  presentation,  focused  history  searches, 
electronic  logbook  (passive  and  active  data  collection),  data/information/knowledge 
amalgamation,  and  collaboration  mechanisms.  (See  CDRL  AO  16,  “System  Design 
Description”).  Design  and  development  time  was  equally  spent  developing,  integrating, 
and  testing  the  components  of  the  technical  data  presentation  and  the  historical  search 
capabilities,  as  well  as  identifying,  down-selecting,  installing,  and  configuring  the 
commercial-off-the-shelf  (COTS)  hardware  and  software  components  necessary  to 
support  the  troubleshooting  environment.  Also  during  this  phase,  the  Field 
Demonstration  Test  (FDT)  plan  was  developed,  validated,  and  approved.  Queries  of 
actual  maintenance  data  sources  were  developed  and  the  data  subsets  were  tested  to 
assure  fitness  for  use  in  the  field  demonstration  test. 

•  Field  Demonstration  Test:  The  third  phase  of  the  program  entailed  conduct  of  the  FDT 

wherein  the  AMIT  capability  was  evaluated  via  a  laboratory-style  experiment  conducted 
in  a  realistic  (flightline  troubleshooting)  environment.  The  test  was  conducted,  the  results 
analyzed,  conclusions  were  drawn,  and  recommendations  were  made.  The  test  was 
constructed  to  directly  compare  the  or  current  AF  flightline  troubleshooting 

methodology  with  the  ‘To-6e”  methodology  that  included  the  AMIT  solution.  The  Team 
developed  test  cases  designed  to  challenge  personnel  in  two  Air  Force  Specialty  Codes 
(AFSC):  Avionics  (2A3X2)  and  Electrical  and  Environmental  (E  &  E)  (2A6X6).  These 
specialties  were  selected  because  successful  troubleshooting  of  Avionics  and  E  &  E 
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systems  consistently  relies  on  “intuitive  leaps”  of  maintainers  and  because  these  systems 
were  found  to  be  both  technically  and  operationally  problematic.  A  mix  of  72  F-16 
aircraft  maintenance  persormel  including  system  specific  Novices,  system  specific 
Experts,  and  non-system  specific  Experts  (i.e.  Crew  Chiefs)  participated  in  the 
experiment. 

1.2  Principal  Results 

Successful  testing  of  the  AMIT  solution  in  a  field  maintenance  environment 
demonstrated  that  the  solution  is  a  militarily-useful  technology.  The  demonstrated  savings  of  45 
to  55  minutes  per  repair  would  result  in  over  47,000  clock  hours  saved,  per  year,  on  the  fleet  of 
F-16  Block  40/42  aircraft.  This  hourly  figure  equates  to  roughly  5.4  aircraft  available  for  service 
per  year.  While  no  direct  metric  exists  to  translate  the  hourly  figure  into  an  8-Hour  Fix  Rate, 
such  an  improvement  would  undoubtedly  be  significant. 

Of  equal  importance,  senior  maintainers  across  the  AF  who  reviewed  the  AMIT 
requirements  and  verified  their  validity  have  repeatedly  expressed  their  support  for  the  AMIT 
program  and  affirmed  the  immediate  need  for  the  solution  in  the  field.  Periodic  briefings  and 
demonstrations  to  AF  senior  leadership  at  the  wing,  command,  and  headquarter  levels  met  with 
equal  approbation.  To  date,  the  program  has  received  unilateral  support  from  the  Air  Combat 
(ACC),  Air  Education  and  Training  (AETC),  and  the  AF  Special  Operations  (AFSOC) 
Commands.  Senior  logistics  leaders  at  Headquarters  AF  (HAF/A4M)  have  seen  demonstrations 
of  the  solution  and  signed  a  Memorandum  of  Understanding  (MOU)  supporting  the  AMIT 
program  and  its  objectives. 
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1.3  Recommended  Course  of  Action 

In  light  of  the  test  results  and  command  support,  the  AMIT  Team  recommends 
transitioning  the  AMIT  concept  and  software  to  ACC  for  final  development  and  initial 
implementation,  with  the  ultimate  goal  of  implementing  the  solution  throughout  the  AF.  The 
Team  believes  the  solution  can  be  fully  developed  and  ready  for  initial  deployment  in  24  months 
or  less,  well  within  the  5  years  stipulated  in  DoDI  5000.2,  “Operation  of  the  Defense  Acquisition 
System.” 
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2  Introduction 

2.1  Aircraft  Maintenance  Intuitive  Troubleshooting 

Flightline  aircraft  maintainers  have  an  excellent  record  of  identifying  and  correcting 
causes  of  system  and  subsystem  failures  and  keeping  highly  complex,  aging  aircraft  flying. 
These  technicians  continue  to  troubleshoot  an  aircraft  until  the  problem  is  isolated;  sometimes 
aided  and  sometimes  hindered  by  current  maintenance  and  logistics  processes  and  procedures  or 
even  the  maintenance  environment  itself.  It  takes  little  analysis  to  determine  that  the  success  or 
failure  of  the  troubleshooting  process  hinges  on  the  technician. 

The  AF  has  placed  great  emphasis  on  enterprise  process  improvements  and  cost  reduction 
programs.  However,  it  has  placed  less  emphasis  on  the  processes  and  systems  supporting  the 
technician  in  making  effective  troubleshooting  decisions  at  the  task  level.  Thus,  while  the 
current  troubleshooting  environment  is  functional,  it  is  often  inefficient.  The  AMIT  program’s 
purpose  was  to  identify  the  factors  (especially  human  factors)  that  contribute  to  inefficiency  in 
troubleshooting  intensive  portions  of  the  maintenance  process,  and  devise  proven  (i.e.,  field 
tested)  means  to  improve  efficiency.  If  those  means  are  demonstrated  to  be  statistically 
significant,  their  introduction  into  the  AF  maintenance  environment  should  have  a  meaningful, 
positive  impact  on  aircraft  availability. 

2.1.1  Terminology 

For  this  effort,  the  AMIT  Team  defined  troubleshooting  as  “a  process  that  includes  all 
activities  related  to  determining  the  correct  repair  action.”  This  included  all  activities  from 
problem  notification  through  repair  verification;  understanding  the  current  discrepancy,  technical 
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data  research,  historical  maintenance  data  research,  determining  aircraft  condition,  and 
communication. 

Efficiency  and  inefficiency  are  inverse  indicators  of  a  technician’s  troubleshooting 
proficiency,  where  proficiency  is  a  vectored  quantity  (i.e.,  a  mathematical  representation  of  a 
physical  phenomenon  where  direction  is  as  important  as  the  value;  “velocity”  and  “force”  are 
two  such  physical  phenomena).  In  other  words,  proficiency  is  a  combination  of  the  time  it  takes 
(troubleshooting  time)  to  make  the  right  decision  (troubleshooting  accuracy). 

The  AMIT  program  did  not  measure  AF  technicians’  troubleshooting  proficiency 
quotients  or  even  attempt  to  develop  a  method  for  doing  so.  Had  that  been  done,  however,  an 
efficiency  distribution  with  an  upper  bound  of  some  practicable  state  of  proficiency  and  a  lower 
bound  of  an  acceptable  state  of  sufficiency  would  result  as  shown  in  Figure  1  (Gott,  S.  P.  1998). 


Figure  1  Relationship  of  Most  Key  Terms  Used  in  the  AMIT  Program 
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A  statistical  analysis  of  this  theoretical  efficiency  distribution  would  yield  a  dividing  line. 
Those  troubleshooters  with  proficiency  quotients  to  the  right  of  the  line  would  be  considered 
Experts  (i.e.,  proficient)  and  those  to  the  left  would  be  considered  Novices  (i.e.,  sufficient)  for 
the  purposes  of  the  AMIT  program.  (NOTE:  In  the  AMIT  program,  an  individual  technician’s 
proficiency  quotient  is  neither  a  reflection  of  his  or  her  fitness  for  the  job  nor  a  measure  of  their 
personal  performance.  The  AMIT  program  simply  used  this  theoretical  construct  of  proficiency 
quotients  to  describe  two  classes  of  test  subjects,  “Novices”  and  “Experts”.) 

As  illustrated  in  Figure  1,  the  troubleshooting  process  entails  three  domains  of  cognition: 
process  knowledge,  strategic  knowledge,  and  system  knowledge.  A  troubleshooter  leaves 
technical  school  with  relatively  more  process  than  system  or  strategic  knowledge. 

Process  knowledge  is  general  in  nature.  The  troubleshooter  knows  the  theory  and 
operation  of  the  major  components  and  subsystems  on  an  aircraft  (e.g.,  jet  engines,  landing  gear 
actuators,  etc.),  how  to  find  and  use  the  technical  documentation,  test  equipment,  and  fault 
isolation  procedures.  The  availability  of  accurate,  current  information  is  critical  in  this  cognitive 
domain  (Gott  S.  P.1989). 

System  knowledge  is  the  knowledge  of  a  specific  type  of  system,  or  set  of  interrelated 
functional  components  on  the  aircraft,  including  design-specific  interactions  and  interfaces  with 
other  systems.  System  knowledge  in  the  Novice  is  minimal,  but  grows  with  on-the-job  exposure 
to  a  myriad  of  details  and  situations.  Experiential  increases  in  process  and  system  knowledge 
result  in  a  more  mature  level  of  strategic  knowledge  (i.e.,  mental  database)  or  reference  model 
of  facts  and  experiences  that  support  better  decision  making.  The  relative  size  of  the  spheres  in 
Figure  1  depicts  the  development  of  the  troubleshooter’s  process,  system,  and  strategic 
knowledge  over  time. 
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At  this  point,  the  troubleshooter’s  personality,  training,  innate  abilities,  and  on-aircraft 
experience  bring  process,  system,  and  strategic  knowledge  into  balance  and  heighten  the 
technician’s  ability  to  synthesize  information,  reducing  task  time.  The  technician’s  proficiency 
quotient  is  now  to  the  right  of  the  average  and  he  or  she  can  be  acknowledged  as  an  expert. 
Unfortunately,  this  corporate  knowledge  will  soon  be  lost  due  to  advancement,  retirement,  or 
separation,  until  another  troubleshooter  completes  his  or  her  pilgrimage  to  proficiency. 

2.1.2  Theory 

The  AMIT  program’s  strategic  goal  was  to  identify  a  set  of  changes  to  the  current  AF 
maintenance  environment  that  would  improve  troubleshooters’  overall  proficiency,  thereby 
increasing  aircraft  availability.  The  team’s  first  challenge  was  to  identify  a  set  of  metrics  that 
establish  a  baseline,  against  which  improvement  could  be  measured.  It  was  evident  from  the 
program’s  outset  that  the  metrics  collected  during  the  field  demonstration  test  (FDT)  were  vital 
to  the  program’s  credibility  in  the  scientific  community  and  critical  to  a  business  case  for 
transitioning  the  solution  into  the  field  if  the  solution  were  successful.  Those  metrics  and  the 
practicability  of  their  collection  would  be  considerations  in  both  the  design  of  the  solution  as 
well  as  the  design  of  the  FDT. 

Troubleshooting  proficiency,  a  function  of  both  task  accuracy  and  task  time,  would  be  an 
ideal  metric.  Plotting  the  average  troubleshooters’  proficiency  quotients  over  time  would  yield  a 
trend  line’s  direction  and  indicate  whether  or  not  changes  in  the  AF’s  maintenance  environment 
were  beneficial  or  detrimental.  In  addition,  its  slope  would  indicate  the  rate  of  change.  A 
positive  and  large  slope  would  indicate  that  the  changes  were  increasing  proficiency;  aircraft 
availability  trend  analyses  should  show  a  corresponding  improvement.  Since  the  AF  does  not 
presently  derive  such  a  metric,  developing  and  validating  a  methodology  for  doing  so  was 
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outside  the  spirit,  scope,  and  resources  of  the  AMIT  program.  The  team  had  to  identify  other 
practicable  metrics  to  quantitatively  demonstrate  any  performance  improvements  afforded  by  the 
solution. 

Since  increased  aircraft  availability  was  the  desired  outcome,  and  the  AF  has  long 
established  data  collection  mechanisms  and  defined  formulae  for  calculating  aircraft  operational 
availability  (Ao),  the  AMIT  team  considered  using  those  measurement  systems  and  metrics  as  a 
barometer  for  the  AMIT  solution’s  success.  However,  a  review  of  the  existing  AF  Ao  metrics 
revealed  that,  for  the  purposes  of  this  experiment,  such  an  analysis  would  include  such  a  large 
number  of  variables  the  program  would  have  to  extend  much  beyond  the  contract’s  period  of 
performance,  while  also  confounding  the  experimental  findings. 

A  third  approach  considered  was  to  measure  the  solution’s  impact  on  recurring  or 
intermittent  problems.  If  the  solution  caused  significant  improvements  in  task  accuracy,  then  the 
number  of  problem  reports  closed  with  a  Can  Not  Duplicate  (CND)  entry,  and  those  determined 
to  be  repeats  (same  problem  on  the  next  flight)  or  recurs  (same  problem  on  the  subsequent  four 
or  five  flights,  depending  upon  aircraft  type)  should  decrease  noticeably.  Here  again,  deriving  a 
viable  test  that  involved  flying  the  same  aircraft  four  or  five  times  under  controlled  conditions 
was  not  practical. 

The  AMIT  team  finally  settled  on  a  3  x  3  x  2  mixed  factorial  experimental  design  that 
would  yield  empirical,  objective  results.  The  test  was  designed  to  include  three  (3)  groups  of  test 
subjects:  system  specific  Novices,  system  specific  Experts,  and  a  non-system  specific  Experts 
(referred  to  as  “Novice,”  “Expert,”  and  “Crew  Chief,”  respectively,  throughout  the  remainder  of 
this  document).  These  groups  completed  maintenance  tasks  of  three  (x  3)  degrees  of  difficulty: 
Simple,  Moderate,  and  Complex,  to  be  accomplished  With  or  Without  (x  2)  the  AMIT  solution. 


II 


Task  Time  and  Task  Accuracy  measurements  would  be  collected  and  the  troubleshooting 
proficiencies  of  the  test  groups  would  be  compared  and  contrasted.  Any  improvements  in 
proficiency  would  become  the  basis  for  predicting  the  improvements  of  larger  population  groups. 
(See  CDRL  A008,  “Demonstration  Test  Plan”) 

During  test  planning  and  preparation,  SMEs  pointed  out  two  separate,  but  equally 
effective,  approaches  to  troubleshooting:  1)  hard  failures  of  discrete  components,  which  lend 
themselves  to  a  procedurally  oriented  fault  isolation  tree  approach,  and  2)  repeat  and  recur 
problems  that  indicate  soft  or  intermittent  failures,  which  favor  a  divide-and-conquer  signal 
tracing  approach.  In  order  to  gauge  the  applicability  of  the  AMIT  solution  to  both 
troubleshooting  approaches,  the  test  included  two  different  subsystems,  one  favoring  each 
troubleshooting  approach. 

2.1. 2.1  Extent  of  Theory 

A  by-product  of  the  AMIT  program  was  the  documented  paradigm  of  the  AF  flight 
maintenance  environment  shown  in  Figure  2.  The  paradigm  provided  a  working  framework 
within  which  the  Team  could  rate  and  prioritize  all  change  opportunities  for  their  potential 
impact. 
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The  AMIT  Solution  Space 


devices  and  are  not  typically  affected  by  human  performance,  the  AMIT  team  left  this  area  to 
those  involved  with  the  practical  applications  of  probability  theory. 

The  z-axis  represents  the  cognitive  dimension  of  maintenance.  Troubleshooters 
synthesize  data  and  information  into  knowledge,  and  make  intuitive  leaps  from  symptoms  to 
solutions  in  this  dimension.  Cyclically,  novices  mature  to  experts  as  they  gain  wisdom  and  that 
wisdom  or  corporate  knowledge  erodes  away  as  the  experts  move  on  with  their  lives  and  careers. 

Thus,  the  AMIT  program’s  problem/solution  space  encompassed  the  entire  AF 
maintenance  paradigm:  doctrine,  organization,  training  and  education,  materiel,  leadership, 
personnel,  and  facilities;  the  physical  work  and  information  processing  environments;  the 
maintainer’s  psychological  makeup,  including  the  way  they  learn  and  think;  their  decision 
making  skills  and  habits;  and  their  character  traits.  All  were  considered  to  be  within  scope 
during  the  program’s  formative  period. 

2.1. 2.2  Assumptions/Constraints 

Because  no  initial  constraints  were  imposed  on  either  the  problem  or  solution  space,  the 
AMIT  Team  had  complete  total  freedom  of  thought  about  what  the  problems  might  be,  what 
their  underlying  causes  might  be,  and  what  solutions  might  affect  positive  changes.  This  mental 
freedom  allowed  the  team  to  “think  outside  the  box”  regarding  investigatory  techniques, 
analytical  approaches,  and  technologies. 

The  program’s  “freedom-of-thought”  environment  was  nurtured  by  the  initial  operating 
assumptions  stated  at  the  AMIT  program’s  outset: 

•  Any  aspect  of  the  current  AF  maintenance  environment  could  be  changed  if  the 
objective  evidence  provided  confidence  in  the  change’s  positive  impact  on  human 
performance  and  the  return  on  investment  justified  the  resource  expenditures. 
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This  assumption  gave  the  team  the  freedom  to  delve  into  any  and  every  aspect  of 
the  current  maintenance  environment  for  its  improvement  potential. 

•  The  AMIT  program’s  efforts  will  result  in  the  transition  of  one  or  more  improved 
troubleshooting  capabilities  into  the  AF’s  System  Development  and 
Demonstration  acquisition  process  and/or  one  or  more  changes  to  the  current 
maintenance  environment.  This  assumption  not  only  bred  an  expectation  of 
success,  but  it  kept  the  team  focused  on  deriving  benefit  to  the  troubleshooter. 
Moreover,  this  expectation  established  the  need  for  efforts  targeted  toward 
successfully  transitioning  the  AMIT  solution  from  the  program’s  outset.  Other 
potential  improvements  in  the  maintenance  process  were  identified,  but  if  they  did 
not  directly  benefit  the  troubleshooter,  they  were  excluded  from  further 
consideration. 

•  No  radical  or  disruptive  changes  will  occur  within  the  maintenance  environment 
before  the  AMIT  solution  is  fully  developed  and  fielded.  If  such  a  change  occurs, 
the  cost  analysis  contained  in  this  document  would  need  to  be  reevaluated  in  light 
of  the  changed  environment. 

2.1.3  Purpose  -  Program  Goals 

In  keeping  with  the  restrictions  upon  6.3  (Advanced  Research)  funds  and  the  DoD  5000 
series  guidelines  for  technology  development,  the  purpose  was  to  identify  an  affordable 
increment  of  militarily-useful  capability,  to  demonstrate  that  capability  in  a  relevant 
environment,  and  describe  how  a  system  can  be  developed  for  production  within  a  short 
timeframe  (normally  less  than  5  years)  (DoDI  5000.2,  f  3.6.7).  In  other  words,  the  AMIT 
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program  had  to  design  and  develop  a  capability  sufficient  for  a  field  demonstration,  without  fully 
developing  the  final  fieldable  solution. 

2.1.4  Defining  Customer  Needs  and  Processes 

Literature  Reviews,  Process  Interviews,  and  Cognitive  Task  Analyses  were  chosen  for 
the  unique  areas  of  insight  they  provided.  The  AMIT  team  believed  that  such  a  combination 
would  paint  a  more  accurate  picture  of  the  maintainers’  thought  processes,  the  maintenance 
environment,  and  the  requirements  for  any  solution  intended  to  increase  a  maintainer’s 
proficiency  than  any  single  technique.  All  three  were  conducted  concurrently,  because  each 
targeted  a  different  outcome:  Literature  Reviews  to  determine  the  state  of  knowledge  and  the 
state  of  technology  as  it  pertained  to  the  problem;  Process  Interviews  towards  discovering  the 
current  maintenance  process;  and  Cognitive  Task  Analyses  to  determine  how  Novice  and  Expert 
maintainers  mentally  approached  troubleshooting  tasks. 

2.1. 4.1  Literature  Review 

A  Literature  Review  (Lit  Review)  was  performed  to  identify  and  scientifically  qualify 
current  and  prototype  technologies  and  research  endeavors  that  might  support  the  AMIT 
solution.  Through  this  Lit  Review,  the  AMIT  team  established  user-based  constraints,  targeted 
key  research  findings  for  application,  and  identified  additional  research  opportunities  for  future 
improvements  to  the  maintenance  environment. 

Lit  Reviews  were  conducted  in  two  parts.  Initially,  the  Team  reviewed  AFRL  program 
reports  in  conjunction  with  Process  Interview  and  Cognitive  Task  Analysis  results  to  establish 
the  Core  Topic  Areas  listed  in  Table  1.  An  extended  Lit  Review  then  sought  and  analyzed 
Department  of  Defense  (DoD)  and  academic  works  in  the  Defense  Technical  Information  Center 
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(DTIC)  and  the  Ohio  Library  and  Information  Network  (OhioLINK)  relevant  to  the  Core  Topic 
Areas. 


Table  1  Core  Topic  Areas  that  Directed  the  AMIT  Team  Literature  Search 


Core  Topic  Areas 

Reference 

Identifiers 

#  Documents 
Reviewed 

#of 

Findings 

Best  Business  Practices 

BBP 

3 

5 

Cognitive  Task  Analysis 

CTA 

13 

28 

Collaboration 

COL 

37 

172 

Decision  Support 

DSS 

22 

100 

Integrated  Human  Interface 

IHI 

3 

13 

Maintenance 

MTN 

24 

71 

Models 

MOD 

4 

18 

Training 

TRN 

5 

30 

Transition 

TRA 

4 

11 

Other 

OTH 

4 

63 

Total  Reviewed 

119 

511 

Documents  chosen  for  both  reviews  were  analyzed  and  reduced  to  brief  statements,  or 
findings,  with  each  expressing  a  single  thought.  The  830  statements,  virtually  verbatim  extracts, 
were  categorized  and  summarized  into  the  63  potential  AMIT  requirements  shovra  in  Figure  3. 
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Figure  3  AMIT  Solution  Requirements  Included  Comprehensive  Maintenance  Processes 
A  full  description  of  the  methods  used  to  build  the  complete  AMIT  requirements 
traceability  matrix  was  provided  in  CDRL  AO  17,  “Systems  Specification.” 

2.1. 4.2  Process  Interviews 

In  order  to  ensure  that  the  AMIT  solution  would  be  “an  increment  of  militarily  useful 
capability,”  the  Team  conducted  a  series  of  process  interviews  to  determine  whether 
organizations  and  practices  were  indeed  uniform  across  the  AF.  In  all,  27  Process  Interviews 
with  subjects  from  ten  different  weapon  systems,  seven  technical  specialties,  and  two  major 
commands  demonstrated  that  the  maintenance  organization  structures  and  processes  were 
consistent  across  different  AF  wings,  with  very  minor  exceptions.  The  minor  exceptions  were 


due  to  the  numbers  of  assigned  maintainers  and  the  type  of  aircraft,  not  to  different  maintenance 
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philosophies,  approaches,  methodologies,  or  practices.  The  interviews  imcovered  some  recurring 
themes  that  are  indicative  of  problem  areas  for  maintainers.  Furthermore,  it  was  determined  that 
on-aircraft  maintenance  has  not  radically  changed  over  the  past  50  years  and  does  not  appear 
likely  to  change  in  the  next  5  years.  All  process  interview  results  were  considered  when  the 
AMIT  requirements  were  defined  (See  CDRL  AO  17,  “System  Specification”). 

2.1. 4.3  Cognitive  Task  Analysis 

In  order  to  understand  the  “intuitive”  nature  of  troubleshooters.  Cognitive  Task  Analysis 
(CTA)  interviews  were  conducted  to  capture  the  thought  processes  that  occur  during  aircraft 
system  troubleshooting  activities.  The  CTA  focused  on  both  Novice  and  Expert  troubleshooter 
thought  processes  and  information  needs  for  cognitively  difficult  or  complex  maintenance  tasks. 
Task  Diagrams  and  Knowledge  Audit  techniques  were  used  as  the  primary  CTA  instruments  for 
this  task.  Task  Diagrams  yielded  a  procedural  perspective  on  maintainer  decision-making,  while 
Knowledge  Audits  provided  insight  into  Expert  troubleshooting  strategies  and  techniques.  (See 
CDRL  017,  “System  Specification,”  for  specifics  on  the  AMIT  CTA.) 

Figure  4  depicts  the  similarities  and  differences  between  Novice  and  Expert  AF  on- 
aircraft  maintainers’  thought  processes  and  awareness  and  usage  trends  of  knowledge  stores. 
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Novice  Resources  Maintenance  Process  Expert  Resources 


Figure  4  Expert  Maintainers  Think  Differently  Than  Novices 

An  analysis  was  completed  after  each  set  of  CTA  interviews,  and  again  when  all  sets 
were  completed.  High-level  themes  within  and  across  interviews  were  identified  using  a  Card 
Sort  method.  The  findings  were  consistent  with,  and  expanded  upon,  those  from  the  Process 
Interviews.  The  CTA  also  identified  three  areas  in  the  maintenance  process  that  were  most 
problematic,  especially  for  Novice  troubleshooters.  These  were  debrief,  problem  duplication, 
and  troubleshooting.  As  with  the  Process  Interviews,  the  CTA  contributed  to  the  requirements 
for  the  AMIT  solution  (See  CDRL  AO  17,  “System  Specification”). 

2.1.5  Design  and  Development 

Whereas  very  few  constraints  were  placed  on  the  AMIT  problem  and  solution  spaces, 
giving  almost  total  freedom  of  thought  during  the  research  and  conceptualization  phases,  the 
resulting  requirements  placed  several  constraints  on  the  AMIT  solution’s  design  and 
development.  The  design  and  development  requirements  posed  a  set  of  often  competing 
institutional,  engineering,  and  human  factors  challenges  as  shown  in  Figure  5.  The  AMIT 
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solution  needed  to  support  multiple  weapon  systems,  be  usable  on  the  flightline,  integrate  easily 
into  existing  AF  maintenanee  support  systems  and  emerging  architectures  such  as  Global 
Combat  Support  System  -  Air  Force  (GCSS-AF),  and  intuitively  improve  troubleshooting  across 
a  range  of  maintainer  skill  levels. 


AF  Infon)13ti0lirSySt6inS  (Maintenance  independent) 


Repair  Databases  IT  Networks  Security  and  Identification  Approved  Hardware ' 

Software  Development  Rules  and  Documentation 

M^tenance  Environment  (Worker  Independent) 

^olhibcaation..-rTdplr7feignme  Parts  mdTtwls 

^rtMx  ~'StrrfbPha^p<!  y.  'grwig'aiiil  Wniklfu 


.^craft  System  Knowledge  (Worker  Dependent) 


Experience  Training  Theory  of  Operations  Task  Complexity 


Testing 

Constraints 


Figure  5  Competing  Constraints  of  the  AMIT  Solution 
The  research  revealed  several  promising  techniques  and  technologies  that  will  likely  have 
a  great  impact  on  the  maintenance  of  complex  systems  and  troubleshooting  one  day,  but  few  that 
were  or  would  be  mature  enough  to  be  incorporated  into  the  AMIT  solution  and  undergo  field 
testing  before  the  conclusion  of  the  program. 

Keeping  focus  on  the  troubleshooter,  the  Team  reevaluated  every  requirement  and 
challenge.  Three  overarching  design  constraints  emerged  during  this  requirements  refinement: 
1)  improve  troubleshooting,  2)  integrate  with  established  AF  systems,  and  3)  transition  easily 
into  the  flightline  maintenance  environment.  The  result  was  a  system  design  description 
integrating  existing  AF  maintenance  data  and  applications,  COTS  software  products,  and  a 
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methodically  designed,  browser-based  user  interface  to  yield  a  job  performance  aid  (JPA)  within 
which  basic  functionalities  of  five  core  capabilities  would  be  co-resident. 

The  five  AMIT  core  capabilities  are:  (1)  Visualization,  (2)  Cognitive  Adaptation,  (3) 
Process  Adaptation,  (4)  Data  Transformation,  and  (5)  Collaboration  (see  Figure  6).  Figure  6  also 
demonstrates  the  relationships  of  the  capabilities  to  the  types  of  knowledge  shovm  in  Figure  1.  A 
full  exposition  of  each  of  the  capabilities,  their  derivation,  and  their  relationships  to  one  another 
was  provided  in  CDRL  AO  16,  “Systems  Design  Description.” 
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AMIT  Core  Capabilities 


Figure  6  Co-resident  Core  Capabilities  in  the  AMIT  Solution 

Thus,  the  AMIT  solution  is  neither  a  new  technology,  nor  a  collection  of  emerging, 
unproven  technologies.  Rather,  it  leverages  existing  data,  maintenance  applications,  and 
information  technologies  in  an  innovative  manner.  The  use  of  technologically  mature  AF 
applications  and  COTS  components  enabled  in  a  full  scale  system  design  will  facilitate  a  final 
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design  and  implementation  phase  of  24  months  or  less,  as  opposed  to  the  5  years  allotted  under 
the  DoD’s  5000-series  acquisition  framework. 

The  Screen  Design  matrix  provided  in  CDRL  A009,  “Software  Design  Description,” 
documents  the  transition  from  capabilities  into  the  complete  software  architecture.  CDRLs 
AO  10,  “Software  Requirements  Specification”  and  AOl  1,  “Software  User  Manual,”  describes  the 
browser-based  user  interface  developed  for  the  AMIT  solution.  The  AMIT  JPA  is  pictured  in 
Figure  7  and  consists  of  the  device  independent  client  solution  installed  on  an  e-tool  qualified  for 
flightline  use.  Ruggedized  laptops  were  selected  because  they  are  already  authorized  for  and  in 
use  on  AF  flightlines. 

COTS  laptops  were  used  as  servers  to  host  the  back  office  portion  of  the  AMIT  solution 

with  which  the  fielded  AMIT  solution 
were  used  to  connect  the  two. 


and  to  simulate  standard  AF  maintenance  support  systems 
interfaced.  Commercial  Wide  Area  Network  technologies 
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Figure  7  AMIT  Job  Performance  Aid  on  Ruggedized  Laptop 


Expert  maintainers  were  consulted  throughout  the  research,  design,  and  development  of 
the  AMIT  solution,  usually  in  a  group  forum.  Aircraft  Maintenance  User  Groups  (AMUGs) 
were  periodically  held  to  ensure  the  team  was  addressing  troubleshooter  needs.  The  AMUGs 
verified  the  requirements,  validated  approaches  and  data,  and  participated  in  Design 
Consideration  Tests  (DCTs)  when  choices  needed  to  be  made  between  two  seemingly  equal 
technical  approaches  or  solutions. 


3  Test  Methods,  Assumptions,  and  Procedures 

The  goal  of  evaluating  the  AMIT  JPA  in  the  maintenance  environment  was  to  emulate,  as 
closely  as  possible,  the  actual  maintenance  environment  while  quantifying  improvements  or 
degradations  in  task  time,  task  accuracy,  or  both.  Measured  improvements  would  illustrate  an 
“increment  of  militarily-useful  capability”  as  “demonstrated  in  a  relevant  environment,”  (see 
CDRL  AO  17,  “System  Specification”)  thereby  satisfying  the  DoDI  5000.2  expectation  of 
Technology  Development  efforts.  The  larger  the  improvement,  the  greater  the  cost-benefit 
associated  with  fielding  the  AMIT  JPA. 

3.1  Hypotheses 

Three  hypotheses  were  developed  for  the  FDT.  Formally  stated,  the  AMIT  hypotheses 

were: 

•  Hypothesis  1:  Maintainers  using  the  AMIT  JPA  will  expend  less  total  time 
troubleshooting  and  performing  research  than  maintainers  who  don’t  use  the 
AMIT  JPA 

•  Hypothesis  2:  Maintainers  using  the  AMIT  JPA  will  make  fewer  errors 
(diagnostic,  unnecessary  removals,  and  fewer  repeat  write-ups)  than  maintainers 
who  don’t  use  the  AMIT  JPA 

•  Hypothesis  3:  Novice  technicians  using  the  AMIT  JPA  will  successfully  and 
efficiently  troubleshoot  difficult  discrepancies  at  or  near  the  same  level  Experts 
not  using  the  AMIT  JPA 
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3.1.1  Scope 

AF  leadership  has  consistently  identified  three  areas  of  great  concern  to  mission 
operations  and  support:  1)  improved  aircraft  availability  (e.g.,  readiness),  2)  reduced  cost,  and  3) 
reduced  deployment  footprint.  Figure  8  illustrates  how  the  Team  perceived  reductions  in  task 
time,  improvements  in  task  accuracy,  and  increases  in  task  proficiency  logically  impact  these 
areas  of  concern.  The  FDT  was  scoped  to  quantify  AMIT  JPA  impacts  on  aircraft  availability 
and  logistics  labor  costs.  Deployment  footprint  was  considered  throughout  the  program,  but 
thoroughly  assessing  and  credibly  quantifying  footprint  impacts  proved  to  be  outside  the  scope 
and  beyond  the  duration  of  the  program. 


Increase  aircraft  availability  by  increasing  troubleshooting  efficiency 


Reduce  logistics  costs  by  reducing  unnecessary  consumption  of  spare  parts 


Minimize  unscheduled  maintenance  effect  on  unit  agility  by  reducing  spare  requirements 


Figure  8  Positive  Effects  of  AMIT  on  Critical  AF  Metrics 


3.2  Testing  Assumptions 

CDRL  A008,  “Demonstration  Test  Plan,”  discusses  the  following  assumptions  pertaining 
to  the  FDT: 
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•  The  test  environment  will  emulate  a  typical  aircraft  maintenance  environment  as 
much  as  possible 

•  Test  data  analysis,  presentations,  and  reports  will  be  concluded  prior  to  contract 
completion 

•  Pilot  Tests  will  be  conducted  prior  to  each  test  phase  in  order  to: 

o  Verify  testing  procedures 

o  Confirm  all  system  components  are  functioning  as  planned 
o  Train  evaluation  team  members  on  system  and  data  collection  procedures 

•  Tests  will  be  conducted  by  AFRL/HEAL  with  support  from  the  contractor  team 
(NCI  and  UDRI) 

•  Data  collection  will  be  conducted  by  AFRL/HEAL  with  technical  and 
administrative  support  provided  by  the  contractor  team 

•  FDT  will  occur  at  Luke  AFB,  AZ,  over  a  4-month  period  (June  to  September, 
2006) 

3.3  Methods 

3.3.1  Experimental  Design 

The  FDT  experimental  design  was  a  3  x  3  x  2  mixed  factorial  with  repeated  measures  on 
one  factor.  The  key  metrics  collected  and  analyzed  were  time  and  errors.  Subjective  feedback 
was  collected,  but  not  analyzed. 

3.3.1. 1  Test  Variables 

The  experiment’s  independent  variables  were  Technician  Experience  Type,  Task 
Difficulty,  and  Tool  Usage.  The  between-groups  variable  of  Technician  Experience  Type 
consisted  of  three  levels  (Novice,  Expert,  Crew  Chief).  The  within-groups  variable  of  Task 
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Difficulty  consisted  of  three  levels  (Simple,  Moderate,  and  Complex).  The  between-groups 
variable  of  Tool  Usage  consisted  of  two  conditions:  without- AMIT  JPA  (As-is)  and  with-AMIT 
JPA  (To-be). 

The  dependent  variables  were  Task  Time  and  Performance  Errors.  Task  Time  ineluded 
total  time  elapsed,  as  well  as  the  time  needed  to  complete  predetermined  task  segments. 
Performance  Errors  were  categorized  as  described  below  and  documented  relative  to  a 
predetermined  optimal  path. 

3.3.1. 2  Apparatus 

Equipment  used  in  the  FDT  can  be  categorized  as  government  furnished  equipment 
(GEE),  government  furnished  data,  contractor  furnished  equipment  (CFE),  or  contractor 
furnished  software.  It  should  be  noted  that  all  software  was  either  COTS  or  developed  in-house 
by  the  AMIT  contractor  team. 

The  GFE  provided  by  the  56*  Maintenance  Group  consisted  of  F-16C  Block  40/42 
aircraft  with  operational  Avionics  and  E  &  E  systems,  all  necessary  Aircraft  Ground  Equipment 
(AGE),  tools,  test  equipment,  technical  order  (TO)  repositories,  and  historical  maintenance  data 
typical  of  an  aircraft  maintenance  unit. 

The  team  evaluated  the  AMIT  hardware  and  related  test  support  hardware  in  the  test 
environment  and  conducted  pilot  tests  prior  to  FDT  to  ensure  proper  operation  and  sufficiency. 
AMIT  JPA  hardware  included  five  laptops,  two  of  which  were  Itronix  Go-Book  III  (i.e., 
ruggedized  laptops  authorized  for  use  as  an  e-tool  on  AF  flightlines)  and  the  remaining  three 
were  standard  laptop  computers:  one  functioned  as  the  AMIT  server,  one  as  the  SME  “ehaf’ 
workstation,  and  the  third  as  the  Integrated  Maintenance  Data  System  (IMDS)  /  E-logbook 
terminal.  All  but  the  IMDS  /  E-logbook  terminal  were  wireless.  The  AMIT  JPA  provided  the 
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participants  all  needed  information  relating  to  the  diagnostic  scenario  and  simulated  access  to 
outside  systems  and  sources. 

The  AMIT-developed  and  COTS  software  resident  on  the  e-tools  constituted  the  AMIT 
JPA,  and  incorporated  the  AMIT-identified  features  and  capabilities  required  to  achieve  the 
program’s  objectives.  The  software  was  installed  and  evaluated  via  a  pilot  test  in  the  test 
environment  prior  to  the  FDT.  In  addition  to  the  AMIT-developed  software,  the  test  set-up 
included  installations  of  TOMCAT  and  Jabber  Chat  Software  for  networking  and  on-line 
chatting  in  the  To-be  test  cases. 

3.3.1 .3  Subjects 

To  ensure  AMIT  applicability  across  specialties,  24  participants  were  chosen  from  each 
of  three  Air  Force  Specialty  Codes  (AFSCs):  Avionics  (2A3X2),  Electrical  and  Environmental 
(2A6X6),  and  Tactical  Aircraft  Maintenance,  also  known  as  “Crew  Chief’  (2A3X3).  Avionics 
specialists  were  assigned  a  series  of  Avionics  repair  tasks,  while  the  Electrical  and 
Environmental  (E  &  E)  specialists  were  assigned  a  series  of  E  &  E  repair  tasks.  Half  the  Crew 
Chiefs  were  assigned  the  Avionics  tasks  and  half  the  E  &  E  tasks.  Thus,  a  total  of  36  test 
subjects  (24  specialists  and  12  Crew  Chiefs)  participated  in  a  given  series  of  repair  tasks.  While 
no  comparison  between  the  Avionics  and  the  E  &  E  specialties  was  conducted,  inclusion  of  both 
was  essential  to  the  experiment’s  integrity  because  of  the  different  troubleshooting 
methodologies  they  invoke.  Avionics  specialists  depend  upon  Built-In-Test/Self  Test  (BIT/ST) 
results  to  identify  their  troubleshooting  strategy  whereas  E  &  E  technicians  largely  depend  upon 
human-observed  and  reported  problems.  While  both  approaches  also  have  numerous 
commonalities,  inclusion  of  members  from  both  groups  was  necessary  for  a  realistic 
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representation  of  the  AMIT  end-user  community.  Additionally,  the  inclusion  of  both  groups 
increased  opportunities  to  extrapolate  results  across  systems  and  airframes. 

3.4  Procedure 

3.4.1  Schedule 

Participant  schedules  were  based  on  an  average  discrepancy  isolation/repair  time  of  no 
more  than  3  hours  per  discrepancy.  The  schedule  (provided  in  CDRL  A008,  “Demonstration 
Test  Plan”)  included  four  3-week  sessions  with  72  total  participants  (24  participants  from  the 
each  of  the  three  specialty  codes). 

As  depicted  in  Table  2,  participants  were  scheduled  for  a  Training  session  (T),  and  three 
subsequent  discrepancy  isolation/repair  sessions  (S),  (SI,  S2,  and  S3).  Three  difficulty  levels  are 
depicted  as  well;  Simple,  Moderate,  and  Complex  (-S,  -M,  -C).  Examples  shown  in  the  “Week 
Days”  columns  indicate  a  typical  activity  schedule  for  a  given  day.  For  example,  the  notation 
T/Sl-S  indicates  the  subject  received  training  (as  denoted  by  7).  SIS  denotes  his/her  first 
session  was  the  Simple  difficulty  level.  As-is  and  To-be  situations  were  scheduled  and  controlled 
by  the  test  team,  and  are  not  depicted  in  the  table.  Each  participant  took  part  in  no  more  than  two 
sessions  per  day,  lasting  no  longer  than  7  hours  per  day.  Maximum  total  time  per  participant  was 
2  duty  days. 
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Table  2  Typical  Session  Schedule  for  Two  AMIT  Test  Subjects 


Avionics  Session 

1 

Week  Days 

Participant 

Number 

Tool 

Technician 

Type 

M 

T 

w 

T 

F 

Week 

1 

1 

Without 

AMIT 

Expert 

S2-M/S3-C 
0700  - 
1300 

Week 

1 

2 

With 

AMIT 

Novice 

T/Sl-M 

1100- 

1500 

S2-C/S3-S 

1300  - 
1800 

Presentation  order  of  the  three  levels  of  Task  Difficulty  (Simple,  Moderate,  and 
Complex)  was  counterbalanced  across  all  conditions  and  identical  for  the  Avionics  and  the 
E  &  E  repetitions.  The  six  possible  orders  were  distributed  evenly  across  the  three  Experience 
Levels  (Novice,  Expert,  and  Crew  Chief)  and  the  two  AMIT  JPA  usage  conditions  (As-is  and  To- 
be).  Additionally,  participants  were  grouped  by  three  so  that  within  a  group,  each  Experience 
Level  was  present,  and  order  of  Task  Difficulty  was  represented  orthogonally,  via  Latin  Square 
methodology.  The  purpose  for  the  counterbalanced  design  was  to  offset  any  potential  effects  due 
to  order,  practice,  or  fatigue. 

3.4.2  Experimental  Tasks 

Six  test  situations  were  developed,  one  for  each  task  difficulty  level  (Simple,  Moderate, 
and  Complex)  and  for  each  system  (Avionics  and  E  &  E).  While  the  actual  discrepancy 
remained  identical  across  AMIT  JPA  usage,  specific  versions  of  each  test  situation  was 
developed  for  AMIT  JPA  usage  conditions.  For  example,  format  requirements  for  logbook  and 
TOs  differed  between  the  with-AMIT  JPA  (To-be)  and  without-AMIT  JPA  (As-is)  conditions. 
Although  the  format  was  different,  every  effort  was  made  to  ensure  that  the  information  content 
and  data  for  both  conditions  remained  parallel. 
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Three  levels  of  Task  Difficulty  were  addressed,  as  follows:  Simple  tasks  involved  a 
maintenance  discrepancy  in  which  the  current  Fault  Isolation  (FI)  tree  indicates  an  acceptable 
solution  via  the  first  or  second  option  listed.  Moderate  tasks  involved  a  maintenance 
discrepancy  in  which  the  solution  was  listed  last  on  the  FI  tree.  For  example,  such  a  situation 
might  have  involved  a  complex  or  lengthy  fault  isolation  procedure  with  wiring  as  the  last  option 
or  a  discrepancy  that  occurred  intermittently.  Complex  discrepancies  involved  one  of  three 
situations:  1)  a  CND  occurrence  where  no  FI  tree  option  was  appropriate,  2)  a  “problem 
persists”  situation,  where  the  Technical  Data  identified  a  corrective  action  if  the  discrepancy  was 
persistent  (Repeat/Recur)  and  all  previous  attempts  at  solving  were  unsuccessful;  and,  3)  an 
"unanticipated"  solution  in  which  the  FI  tree  identified  via  the  fault  code  did  contain  the  solution. 

Each  test  situation  is  listed  below  and  is  labeled  based  on  the  system  affected.  The  list 
includes  the  fault  code  mapping  to  the  fault  isolation  information,  a  sample  write-up  that  would 
be  found  in  the  aircraft  forms,  method  and  location  of  problem  insertion,  and  the  targeted 
solution.  The  six  situations  used  are: 

•  Avionics  Situation  1  —  Radar  Altimeter 

o  Fault  Code  9462XD  (Simple) 

o  Aircrew  Write-up:  Radar  Altimeter  (HALT)  inoperative 
o  Insertion  Method:  Pull  Circuit  Breaker  3943 CB71 
o  Location  of  Problem  Insertion:  Door  2204 
o  Targeted  Solution:  signal  data  converter  (SDC) 

•  Avionics  Situation  2  —  Fire  Control  Radar 

o  Fault  Code  946 IXD  (Moderate) 
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o  Aircrew  Write-up:  Fire  Control  Radar  shut  down  in  flight.  Would  not 
reset. 

o  Insertion  Method:  946 1K2  Clip  pin  X 
o  Location  of  Problem  Insertion:  Panel  2206 
o  Targeted  Solution:  946 1K2  relay 

•  Avionics  Situation  3  —  Fire  Control  Radar 

o  Fault  Code  9461 AJ  (Complex) 

o  Aircrew  Write-up:  Had  MFL  093  and  1 12  in  flight.  Would  not  reset, 
o  Insertion  Method:  N/A  —  Problem  can  not  be  duplicated  on  ground 
o  Location  of  Problem  Insertion:  N/A 
o  Targeted  Solution:  Absolute  pressure  regulator 

•  E  &  E  Situation  1  —  External  Power 

o  Fault  Code  2440MH  (Simple) 

o  Aircrew  Write-up:  During  quick  turn,  aircraft  would  not  accept  external 
power 

o  Insertion  Method:  Relay  2441 K1  Clip  pin  A 1 
o  Location  of  Problem  Insertion:  Panel  3312  and  3308 
o  Targeted  Solution:  244 IKl  relay 

•  E  &  E  Situation  2  —  Air  Conditioning 

o  Fault  Code  21 60BH  (Moderate) 

o  Aircrew  Write-up:  Air  Conditioning  goes  full  cold  in  both  Auto  and 
Manual 

o  Insertion  Method:  2 1 6 1 K2  Clip  pin  A3 
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o  Location  of  Problem  Insertion:  Panel  2101 


o  Targeted  Solution:  Cabin  temperature  control  valve 
•  E  &  E  Situation  3  —  Air  Conditioning 
o  Fault  Code  2160XF  (Complex) 

o  Aircrew  Write-up:  Repeat,  defog  stayed  on  all  the  time 
o  Insertion  Method:  2161K1  Clip  pin  A2 
o  Location  of  Problem  Insertion:  Panel  2101 
o  Targeted  Solution:  2161 K1  relay 

3.4.3  Data  Collection  Instruments 

This  section  describes  the  equipment  and  techniques  used  by  the  FDT  team  to  collect 
performance  data  and  subjective  feedback. 

3.4.3.1  Measuring  Task  Time 

Using  a  stopwatch  total  Task  Times  were  collected,  including  individual  times  associated 
with  specific  processes  and/or  sub-processes.  Task  Time  was  measured  in  terms  of  overall  time 
elapsed  from  beginning  (fault  ID)  to  end  (correct  fix  identified  via  ops  check),  in  addition  to 
predetermined  task  segments.  Examples  of  task  segments  included  time  spent  collaborating  with 
an  SME,  searching  through  the  TOs,  troubleshooting,  and  conducting  searches  of  historical 
documentation.  Certain  task  segments  were  assigned  standard  times;  for  example,  estimated 
travel  time  between  aircraft  and  hangar,  and  time  to  remove  and  install  components. 

This  granular  collection  and  analysis  of  time  data  allowed  quantification  of  AMIT  JPA 
impacts  at  the  process  and  sub-process  levels.  Comparisons  between  To-be  and  As-is  conditions 
reflect  Task  Time  performance  improvements  afforded  by  the  AMIT  JPA.  As  described  in  the 
Results  section,  this  comparison  was  conducted  across  the  entire  task,  as  well  as  between  sub- 
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tasks.  Also,  by  grouping  like  steps  together,  such  as  collaboration  or  historical  research,  analysis 
identified  the  effects  of  individual  capabilities  across  the  complete  maintenance  process.  This 
approach  also  allov^ed  the  optional  exclusion  of  time  contributions  outside  the  scope  of  the  FDT, 
such  as  repair  time  or  part  acquisition  time,  which  for  purposes  of  the  test,  were  given  typical 
constant  values. 

Figure  9  depicts  the  granular  time  collection  strategy,  spanning  the  overall  maintenance 
process.  It  identifies  the  individual  processes  impacted  by  the  AMIT  JPA  and  denotes  whether 
each  process  was  measured  during  the  FDT  or  assigned  a  standard  time.  If  the  Repair 
Verification  failed  after  the  first  pass  through  the  process,  the  four  processes  spanned  by  the  “1 
to  n  cycles”  bar  were  looped.  That  is,  the  maintainer  was  permitted  to  start  over  at  “Time  to 
Isolate.”  The  number  of  “loops”  as  well  as  the  times  for  each  process  within  each  loop  were 
collected  and  analyzed  to  identify  AMIT  JPA  impacts. 
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Figure  9  FDT  Task  Segments  and  Measurement  Approach 
Collaboration  and  historical  maintenance  data  research  are  important  strategies  for 
understanding  difficult  maintenance  problems.  To  accomplish  collaboration  or  historical 
research  in  the  As-is  environment,  the  maintainer  must  physically  travel  to  an  “access  point”  for 
collaboration  or  historical  data  research  (e.g.,  phone,  e-mail,  in  person  for  collaboration, 
logbook,  or  IMDS  terminal).  This  would  have  required  him/her  to  secure  the  aircraft  prior  to 
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travel  and  prepare  the  aircraft  upon  return;  therefore,  these  times  were  included  as  part  of  the 
total  time  for  the  function.  Multiple  occurrences  of  such  actions  were  allowed  and  included  as 
part  of  the  analysis.  Figure  10  illustrates  the  means  by  which  the  granular  data  collection 
strategy  allowed  comparison  of  collaboration  or  historical  research  between  the  To-be  and  As-is 
conditions. 
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Figure  10  Collaboration  and  Historical  Research  as  Part  of  Overall  Repair  Time 


3.4.3.2  Measuring  Task  Accuracy 

Using  an  optimum  solution  set  developed  and  validated  for  each  discrepancy,  maintainer  actions 
were  logged  and  evaluated  for  accuracy.  Two  categories  of  decision  errors  were  created  for  the 
FDT  based  upon  the  underlying  cause  for  the  incorrect  decision.  System  Presentation  errors 
were  those  caused  by  either  the  AMIT  JPA  or  existing  technical  data  presenting  incorrect  or 
incomplete  information  to  the  test  subject.  User  Interpretation  errors  were  those  where  the  test 
subject  misunderstood,  misapplied,  or  simply  did  not  absorb  the  information  presented  by  the 
AMIT  JPA  or  existing  technical  data. 

Maintainers  participating  in  the  FDT,  however,  exceeded  expectations  with  regard  to  the 
original  definition  of  Task  Accuracy.  The  concept  of  an  “error”  was  initially  defined  as  “a  repair 
decision  not  consistent  with  the  optimum  repair  solution,”  with  a  cause  rooted  at  the  user  and/or 
system  level.  These  instances,  however,  were  rarely  observed.  Therefore,  task  accuracy  was 
redefined  to  include  either  Critical  Errors  or  Non-critical  Deviations  from  the  optimum  solution. 
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Critical  Errors  were  those  leading  directly  to  an  incorrect  repair  (e.g.,  using  the  wrong  TO  or 
performing  the  wrong  repair).  Non-critical  Deviations  were  instances  in  which  the  technician 
omitted  an  ops  check  or  a  sub  task  listed  in  the  tech  data  (e.g.,  conducting  Inertial  Navigation 
System  alignment  in  the  Fire  Control  Radar  tasks),  but  nonetheless  successfully  completed  the 
repair. 
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4  Test  Results 

4.1  Presentation  of  Results 

The  FDT  comprised  two  separate,  parallel  tests:  One  addressed  Avionics-related  systems 
and  the  second  addressed  E  &  E  systems.  Direct  statistical  comparisons  between  the  two  were 
never  platmed,  but  the  results  for  both  are  provided  concurrently,  so  the  reader  may  make 
notional  comparisons  in  the  patterns  of  results  contained  within  each  specialty. 

4.1.1  Quantitative  Results 

Statistical  Analyses  were  accomplished  via  an  Analysis  of  Variance  (ANOVA),  or  F-Test 
statistic.  When  described  in  the  body  text  of  this  report,  data  presentation  uses  the  following 
format:  V{dfv,dfe)  =f  (p  =  j?)  where  djv  is  the  degree  of  freedom  associated  with  the  variable,  dfe 
is  the  degree  of  freedom  in  the  error  term,  f  is  the  numeric  value  of  the  f  ratio,  and  p  is  the 
measured  alpha  value,  indicating  the  probability  that  statistical  significance  was  due  to  chance  or 
sampling  error.  Post  hoc  t-test  statistical  analyses  were  performed,  where  appropriate.  Results 
were  categorized  by  system  (Avionics  and  E  &  E)  and  measure  (Time  and  Errors).  Tables  of 
descriptive  statistics  and  ANOVA  summary  tables  are  provided  within  each  section. 

4.1. 1.1  Task  Time 

The  mean  total  task  time  for  each  Avionics  test  scenario  is  provided  in  Table  3  and 
Table  4  provides  the  results  of  the  ANOVA  for  the  Avionics  condition  task  time  analysis. 
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Table  3  Descriptive  Statistics  for  Avionics  Task  Completion  Times 


Task  Completion  Time  (Minutes)  -  Avionics 

Experience 

Difficulty 

Grand  Average 
(Usage  by  Experience) 

Simple 

Moderate 

Complex 

As  Is 

95.95 

119.77 

218.19 

144.64 

Crew  Chief 

96.01 

124.13 

235.96 

152.04 

Novice 

105.66 

110.88 

247.05 

154.53 

Expert 

86.18 

124.31 

171.56 

127.35 

To  Be 

55.00 

72.61 

181.98 

103.20 

Crew  Chief 

57.14 

89.29 

208.39 

118.27 

Novice 

59.58 

74.93 

187.25 

107.25 

Expert 

48.30 

53.63 

150.31 

84.08 

Grand  Average 
(Difficulty): 

75.48 

96.19 

200.09 

Table  4  Avionics  Task  Completion  Time  ANOVA  Summary  Table 


Variable 

Sum  of  Squares 

Df 

Mean  Square 

F 

P 

Tool  Usage  (T) 

46361.138 

1 

46361.138 

17.765 

.001 

Experience  (E) 

18224.528 

2 

9112.264 

3.492 

.043 

TxE 

866.912 

2 

433.456 

.166 

>.05 

Error 

7 829 1.95 1 

30 

2609.732 

Difficulty  (D) 

321004.747 

2 

160502.373 

99.816 

<.001 

DxT 

543.082 

2 

271.541 

.169 

>.05 

DxE 

13043.697 

D 

3260.924 

2.028 

>  .05 

D  X  T  X  E 

4309.008 

D 

1077.252 

.670 

>  .05 

Error 

96479.053 

60 

1607.984 

As  hypothesized,  the  main  effect  of  Tool  Usage  was  significant  for  the  Avionics  task 
scenarios  (F(l,30)  =  17.77,  p  =  .001).  Participants  using  the  AMIT  JPA  successfully  completed 
the  repair  process  roughly  41  minutes  faster  than  those  who  used  available  maintenance 
resources. 
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The  main  effect  of  Task  Difficulty  was  significant  for  the  Avionics  task  scenarios 
(F(2,60)  =  99.82,  p  <  .001).  Post  hoc  pairwise  comparisons  indicated  that  the  Difficult  condition 
yielded  greater  times  to  complete  than  the  Moderate  condition  (t(35)  =  9.40,  p  <.001),  and  the 
Moderate  condition  yielded  greater  times  to  complete  than  the  Simple  condition  (t(35)  =  2.55,  p 
<.01). 

No  significant  interactions  were  observed  for  Task  Time  in  the  Avionics  task  condition. 
The  mean  total  Task  Time  for  each  E  &  E  test  scenario  is  provided  in  Table  5. 

Table  6  provides  the  results  of  the  ANOVA  for  the  E  &  E  condition  Task  Time  analysis. 


Table  5  Descriptive  Statistics  for  E  &  E  Task  Completion  Times 


Task  Completion  Time  (Minutes)  -  E  &  E 

Experience 

Difficulty 

Grand  Average 
(Usage  by  Experience) 

Simple 

Moderate 

Complex 

As  Is 

90.45 

319.18 

148.30 

185.05 

Crew  Chief 

97.42 

324.44 

135.48 

185.78 

Novice 

85.51 

316.49 

119.94 

173.98 

Expert 

88.41 

316.61 

189.49* 

198.17 

To  Be 

51.16 

284.89 

72.55 

136.26 

Crew  Chief 

52.70 

291.33 

93.29 

145.77 

Novice 

52.73 

279.36 

65.68 

132.59 

Expert 

48.05 

283.97 

58.70 

130.24 

Grand  Average 
(Difficulty): 

70.80 

302.03 

110.43 

*Reflects  higher  than  expected  value  due  to  one  outlier.  See  section  4.2.2. 1 . 
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Table  6  £  &  E  Task  Completion  Time  ANOVA  Summary  Table 


Variable 

Sum  of  Squares 

Df 

Mean  Square 

F 

P 

Tool  (T) 

66896.356 

1 

66896.356 

41.257 

<.001 

Experience  (E) 

333.339 

2 

1666.669 

1.028 

>.05 

TxE 

4459.003 

2 

2229.502 

1.375 

>  .05 

Error 

48644.170 

30 

1621.472 

Difficulty  (D) 

1100987.085 

2 

550498.542 

384.087 

<.001 

DxT 

9220.059 

2 

4610.030 

3.216 

.047 

DxE 

3801.651 

4 

950.413 

.663 

>.05 

D  X  T  X  E 

9645.289 

4 

2411.322 

1.682 

>.05 

Error 

85995.851 

60 

1433.264 

As  hypothesized,  the  main  effect  of  Tool  Usage  was  significant  for  the  E  &  E  task 
scenarios  (F(l,30)  =  41.26,  p  <  .001).  Similar  to  the  Avionics  test,  participants  using  the  AMIT 
JPA  successfully  completed  the  repair  process  roughly  50  minutes  faster  than  those  who  used 
available  maintenance  resources. 

The  main  effect  of  Task  Difficulty  was  significant  for  the  E  &  E  task  scenarios  (F(2,60)  = 
384.09,  p  <  .001).  Post  hoc  pairwise  comparisons  indicated  that  the  Moderate  condition  yielded 
greater  task  times  than  the  Difficult  condition  (t(35)  =  18.66,  p<.001),  and  the  Difficult  condition 
yielded  greater  task  times  than  the  Simple  condition  (t(35)  =  3.28,  p<.01).  No  other  main  effects 
were  found  to  be  statistically  significant. 

The  interaction  of  Tool  Usage  x  Task  Difficulty  was  marginally  significant  (F(2,60)  = 
3.22,  p  =  .047).  Figure  1 1  illustrates  the  relationship  between  Tool  Usage  and  Task  Complexity. 
The  difference  between  As-is  and  To-be  completion  times  is  greatest  in  the  Complex  task 
condition;  however,  post  hoc  analysis  revealed  the  strongest  effect  of  Tool  Usage  for  the  Simple 
(t(34)  -  6.34,  p  <.00 1)  and  Moderate  tasks  (t(34)  =  6.19,  p  <.00 1);  and  a  slightly  weaker  but  still 
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significant  effect  of  Tool  Usage  for  the  Complex  task  (t(34)  =  3.41,  p  <.01).  This  was  likely  due 
to  increased  variability  in  As-is  and  To-be  Task  Times  in  the  Complex  task  condition,  as  depicted 
by  the  vertical  error  bars. 

Tool  Usage  x  Task  Difficulty 


350 


Simple  Moderate  Complex 

Task  Difficulty 


Figure  11  Tool  Use  x  Task  Difficulty  Interaction  on  Task  Time 

4.1. 1.2  Task  Accuracy 

Using  the  error  definition  methodology  described  in  section  3,  no  “System  Incomplete” 
or  “System  Incorrect”  error  categories  occurred  during  the  FDT.  All  observed  errors  fell  under 
the  predefined  categories  of  “User  Incomplete”  and  “User  Incorrect.”  In  lieu  of  error  data 
oriented  toward  pinpointing  the  root  cause  as  the  system  and/or  the  user,  errors  were  reclassified 
more  simply  as  Critical  Errors  or  Non-critical  Deviations,  as  noted  in  3. 4. 3. 2  above. 

Six  Critical  Errors  occurred  during  the  Avionics  task;  all  occurred  in  the  As-is  condition 
with  five  erroneous  navigations  through  the  paper  TO.  The  remaining  Critical  Error  occurred 
when  a  maintainer  intentionally,  but  incorrectly,  chose  a  repair  not  prescribed  by  the  appropriate 
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FI  TO.  Because  the  total  number  of  Critical  Errors  were  minimal  across  all  participants  and 
conditions,  statistical  analysis  was  not  conducted  on  Critical  Error  data. 

For  Non-critical  Deviations,  technicians  using  the  AMIT  JPA  deviated  less  often  from 
the  optimum  resolution  (i.e.,  made  fewer  Non-critical  errors)  as  shown  in  Table  7.  Nearly  all 
Non-critical  Deviations  involved  the  technician  omitting  a  procedure  in  the  maintenance  process, 
such  as  fault  verification  or  a  specific  operations  check.  An  unusual  condition  occurred  when 
Expert  technicians  using  the  AMIT  JPA  made  more  Non-critical  Deviations  when  completing 
the  difficult  repair  task  than  Expert  technicians  conducting  the  same  repair  in  the  As-is  condition. 

Contrary  to  hypothesis,  the  main  effect  of  Tool  Usage  was  non-significant  for  the 
Avionics  task  scenarios  (F(l,30)  =  1.46,  p  >  .05)  as  shown  in  Table  8.  Participants  using  the 
AMIT  JPA  did  not  commit  significantly  fewer  Non-critical  Deviations  than  those  who  used 
currently  available  maintenance  resources. 

The  main  effect  of  Task  Difficulty  was  significant  for  the  Avionics  task  scenarios 
(F(2,60)  =  88.83,  p  <  .001).  Post  hoc  pairwise  comparisons  indicated  that  the  Complex  condition 
yielded  more  non-critical  deviations  than  the  Moderate  and  Simple  conditions  (t(34)  =  12.54, 
p<.001),  and  the  Moderate  condition  yielded  more  Non-critical  Deviations  than  the  Simple 
condition  (t(34)  =  12.91,  p<.001). 

No  significant  interactions  were  observed  for  errors  in  the  Avionics  task  condition. 
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Table  7  Descriptive  Statistics  for  Avionics  Non-critical  Deviations 


Non-critical  Deviations  -  Avionics 

Experience 

Difficulty 

Grand  Average 
(Usage  by  Experience) 

Simple 

Moderate 

Complex 

As  Is 

0.44 

2.56 

3.67 

2.22 

Crew  Chief 

0.83 

2.33 

3.83 

2.33 

Novice 

0.00 

2.50 

4.50 

2.33 

Expert 

0.50 

2.83 

2.67 

2.00 

To  Be 

0.28 

2.00 

3.56 

1.94 

Crew  Chief 

0.17 

2.33 

3.83 

2.11 

Novice 

0.50 

1.50 

3.17 

1.72 

Expert 

0.17 

2.17 

3.67 

2.00 

Grand  Average 
(Difficulty); 

m 

2.28 

3.61 

Table  8  Avionics  Non-critical  Deviations  ANOVA  Summary  Table 


Variable 

Sum  of  Squares 

Df 

Mean  Square 

F 

P 

lool  (T) 

3.000 

1 

3.000 

1.462 

>.05 

Experience  (E) 

1.556 

2 

.778 

.379 

>.05 

TxE 

2.889 

2 

1.444 

.704 

>.05 

Error 

61.556 

30 

2.052 

Difficulty  (D) 

202.667 

2 

101.333 

88.831 

<.001 

DxT 

.889 

2 

.444 

.390 

>.05 

DxE 

5.111 

4 

1.278 

1.120 

>.05 

DxTxE 

7.556 

■1 

1.889 

1.656 

>.05 

Error 

68.444 

60 

1.141 

Eighteen  Critical  Errors  were  observed  in  the  E  &  E  task  scenarios.  All  Critical  Errors 
occurred  in  the  As-is  condition.  Five  errors  were  incorrect  navigation  through  the  paper  based 
TO.  Four  were  the  result  of  entering  completed  session  information  into  the  e-logbook  against 
the  wrong  aircraft.  The  remaining  nine  errors  occurred  when  a  technician  intentionally,  but 
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incorrectly  chose  a  repair  not  prescribed  by  the  appropriate  FI  TO.  As  with  the  Avionics  task, 
the  total  number  of  Critical  Errors  were  minimal  across  all  participants  and  conditions;  therefore, 
statistieal  analysis  was  not  conducted  on  Critical  Error  data. 

Average  Non-eritieal  Deviations  for  the  E  &  E  tasks  are  depieted  in  Table  9.  As 
hypothesized,  the  main  effect  of  Tool  Usage  was  significant  for  the  E  &  E  task  seenarios 
(F(l,30)  =  7.80,  p  <  .01)  as  shown  in  Table  10.  Participants  using  the  AMIT  JPA  eommitted 
fewer  Non-critieal  Deviations  than  those  who  used  available  maintenance  resourees. 

No  significant  interactions  were  observed  for  the  measure  of  Non-critical  Deviations. 


Table  9  Descriptive  Statistics  for  E  &  E  Non  critical  Deviations 


Non-critical  Deviations  -  E  &  E 

Difficulty 

Simple 

Moderate 

Complex 

Grand  Average 
(Usage  by  Experience) 

Usage  by  Experience 

As  Is 

0.61 

0.89 

0.39 

0.63 

Crew  Chief 

0.67 

0.67 

0.17 

0.50 

Novice 

0.33 

0.67 

0.00 

0.33 

Expert 

0.83 

1.33 

1.00 

1.06 

To  Be 

0.11 

0.28 

0.22 

0.20 

Crew  Chief 

0.00 

0.33 

0.17 

0.17 

Novice 

0.17 

0.33 

0.17 

0.22 

Expert 

0.17 

0.17 

0.33 

0.22 

Grand  Average 
(Difficulty): 

0.36 

0.58 

0.31 
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Table  10  E  &  £  Non-Critical  Deviations  ANOVA  Summary  Table 


Variable 

Sum  of  Squares 

Df 

Mean  Square 

F 

P 

Tool  (T) 

4.898 

1 

4.898  7 

7.802 

.009 

Experience  (E) 

2.722 

2 

1.361 

2.168 

>.05 

TxE 

2.463 

2 

1.231 

1.962 

>.05 

Error 

18.833 

30 

.628 

Difficulty  (D) 

1.556 

2 

.778 

2.979 

>.05 

DxT 

.963 

2 

.481 

1.844 

>.05 

DxE 

.556 

4 

.139 

.532 

>.05 

D  X  T  X  E 

.593 

4 

.148 

.567 

>.05 

Error 

15.667 

60 

.261 

4.1. 1.3  Task  Proficiency  -  Novice  vs.  Expert  Performance 

Hypothesis  3  stated,  for  complex  tasks,  performance  of  Novice  maintainers  using  the 
AMIT  JPA  would  approach  that  of  Experts  using  current  methods.  Test  results  confirm  that, 
when  completing  the  Complex  task.  To-be  Novice  performance  approached  or  exceeded  As-is 
Expert  performance.  Four  one-tailed,  independent  sample  t-tests  were  conducted  for  the 
combinations  of  specialty  (Avionics,  E  &  E)  and  performance  measure  (Task  Time,  Non-critical 
Deviations),  in  the  Complex  task  condition.  Performance  measures  of  the  To-be  Novice  were 
not  significantly  different  from  the  As-is  Expert  for  Avionics  completion  time  (t(10)=.74  p  > 
.05),  Avionics  Non-critical  Deviations  (t(10)=.38,  p  >  .05),  and  E  &  E  Non-critical  Deviations 
(t(10)=1.27,  p  >  .05).  For  E  &  E  task  completion  times.  Novice  performance  using  the  AMIT 
JPA  was  significantly  shorter  than  Expert  times  in  the  As-is  (i.e.  without  the  AMIT  JPA) 
condition,  for  the  Complex  task  (t(10)=2.07,  p  <  .05). 
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4.1.2  Qualitative/Subjective  Results 

Participants  were  given  a  Likert  scale  questionnaire  regarding  their  use  of  information 
sources  to  help  prepare  for  or  accomplish  a  repair  prior  to  starting  their  first  test  scenario.  The 
information  sources  included:  Logbook,  IMDS,  TOs,  communication  with  other  maintainers, 
technical  representatives,  and  management-level  maintenance  personnel.  On  a  seale  of  0 
(“never”)  to  5  (“often”),  participants  were  asked  to  rate  the  frequency  of  interaction  with  each  of 
the  above  information  resources. 

After  completing  all  maintenance  tasks,  participants  were  given  two  short  (one  page) 
questionnaires.  The  first  asked  them  to  compare  the  difficulty  of  the  following  task  elements 
versus  everyday  maintenance  tasks:  Research,  Collaboration,  TO  Access,  Logbook  Use,  Media, 
and  Transport  of  Media  to/at  the  Aircraft.  The  second  asked  them  to  rate  the  “value  added”  by 
the  AMIT  JPA  in  each  of  the  same  task  elements. 

4.1. 2.1  Resource  Use 

Results  for  the  Avionics  group  indicates  both  Novice  and  Expert  maintainers  tended  to 
consult  with  other  maintainers  to  help  with  a  difficult  repair.  Novices  were  less  likely  to  contact 
a  technical  representative,  while  neither  group  indicated  they  contacted  management-level 
maintenance  personnel  for  assistance.  Consultations  were  usually  eonducted  face-to-face  rather 
than  by  telephone  or  electronic  means.  All  technicians  indicated  a  likeliness  to  prepare  for  a  job 
using  IMDS,  historical  logbook  information,  and  in-depth,  preparatory  study  of  teehnical  orders. 
The  findings  are  consistent  with  those  of  the  CTAs  discussed  in  section  2. 1.4. 3. 

Results  for  the  E  &  E  group  indicates  that  both  Novices  and  Experts  tend  to  consult  with 
other  maintainers  to  help  with  a  difficult  repair,  but  are  less  likely  to  contaet  technical 
representatives  or  management-level  maintenance  personnel  for  assistance.  Such  contact  is 


48 


generally  face-to-face,  rather  than  by  telephone  or  electronic  means.  Experts  indicated  a  greater 
likeliness  to  prepare  for  a  job  with  thorough  research  of  IMDS  and  historical  logbook 
information  and  in-depth  review  of  TOs. 

4.1. 2.2  Comparison  of  Difficulty 

After  completing  all  simulated  maintenance  tasks,  participants  were  asked  to  rate  the 
difficulty  of  Research,  Collaboration,  TO  Access,  Logbook  Use,  Media,  and  Transport  of  Media 
to/at  the  Aircraft  as  compared  to  everyday  maintenance  tasks.  The  scale  ranged  from  -3  (more 
difficult)  to  +3  (less  difficult),  with  “0”  indicating  no  difference  in  difficulty. 

Those  using  the  AMIT  JPA  felt  the  simulated  maintenance  task  was  less  difficult  than 
those  using  the  current  maintenance  resource  set.  While  all  AMIT  JPA  users  ranked  the  tasks  as 
easier  than  everyday  tasks.  Novices  gave  the  To-be  condition  stronger  “easy”  rankings  than 
Experts,  across  all  categories.  Crew  Chiefs’  responses  were  mixed  with  regard  to  category;  for 
Research,  Collaboration,  Logbook,  and  Media  Transport,  their  responses  were  similar  to  those 
of  the  Novices.  However,  for  Accessibility  and  General  Use  of  Media,  the  responses  were 
similar  to  those  of  the  system  Experts.  Average  ratings  for  participants  in  the  As-is  condition  fell 
near  1.0  for  all  categories,  indicating  “slightly  less  difficult.”  In  the  To-be  condition,  average 
Novice  ratings  of  difficult  ranged  from  1.8  to  2.6,  while  Expert  ratings  of  difficulty  comparison 
ranged  between  1.0  and  2.0.  In  the  As-is  condition,  all  experience  levels  indicated  that 
participating  in  the  study  was  no  less  or  more  difficult  than  everyday  work,  while  in  the  To-be 
condition,  less  experienced  technicians  gave  AMIT  more  extreme  “easy”  ratings  than  did 
experienced  technicians. 

The  E  &  E  pattern  of  responses  differed  from  the  Avionics  tasks.  For  example, 
technicians  who  performed  the  E  &  E  tasks,  regardless  of  whether  they  used  the  AMIT  JPA, 
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rated  difficulty  of  the  simulated  tasks  as  easier  than  actual.  A  potential  explanation  of  this  result 
may  be  that  many  of  the  common  E  &  E  sub-tasks  (i.e.,  lengthy  operation  checks  and  repairs) 
were  not  required  for  the  FDT.  One  of  the  selection  criteria  for  the  test  scenarios  was  length  of 
subtasks,  because  they  often  increased  the  total  test  time,  but  did  not  provide  any  data  pertinent 
to  the  AMIT  JPA.  This  was  an  experimental  design  decision  based  on  time  constraints  and  the 
goal  of  the  project. 

4.1. 2.3  Value  Added 

After  completing  all  simulated  maintenance  tasks,  participants  were  asked  to  rate  the 
“value  added”  of  the  AMIT  JPA  for  the  following  tasks:  Research,  Collaboration,  TO  Access, 
Logbook  Use,  Media,  and  Transport  of  Media  to/at  the  Aircraft.  The  scale  ranged  from  +3 
(High  value  added)  to  -3  (High  value  subtracted)  with  “0”  indicating  no  difference.  All  but  two 
of  the  36  Avionic  responses  fell  between  +2  and  +3,  the  only  significant  differences  in  responses 
between  groups  involved  As-is  Novice  and  Expert  specialists  rating  for  Collaboration.  While 
both  groups  gave  positive  value  added  ratings  for  this  capability,  their  average  rating  was  lower 
than  the  Crew  Chiefs’.  All  ratings  for  value  added  except  Collaboration  fell  at  +2.6  to  +2.8  out 
of  a  possible  +3.0. 

E  &  E  data  revealed  a  similar  pattern.  Collaboration  was  the  low  point,  but  still  greater 
than  2  in  all  groups  except  To-be  Novice  (1.8). 

4.1. 2.4  Testimonials/Interviews 

Participants  were  asked  to  comment  on  a  variety  of  AMIT  JPA  capabilities  and  features. 
Participant  feedback  was  resoundingly  positive.  Maintainers  liked  the  automatic  collection  of 
logbook  data  and  the  focused  history  searches.  They  specifically  indicated  that  the  Line 
Replaceable  Unit  (LRU)  history  feature  would  be  of  great  help.  Another  favorite  was  the 
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focused  TO  search  feature,  which  went  beyond  electronic  format  by  providing  the  ability  to 
automatically  navigate  to  the  appropriate  location  in  the  TO  library  based  on  current  fault  code. 

Feedback  on  the  FI  history  tool  was  mixed;  several  cultural  issues  may  exist  regarding 
feedback  on  this  feature.  Further  elaboration  is  provided  in  the  Results  Discussion  section  (4.2). 

Response  to  the  collaboration  capability  was  also  mixed— although  not  all  of  the 
participants  were  convinced  it  would  always  work,  especially  when  attempting  to  contact  the 
aircrew,  none  particularly  disliked  it. 

When  asked  if  using  AMIT  would  help  make  the  Novice  a  better  troubleshooter  more 
quickly,  the  majority  of  participants  responded  “yes.”  Likely  reasons  for  this  response  included 
instant  access  to  the  correct  TO  information,  having  access  to  path-to-resolution  information,  and 
the  ability  to  look  up  LRU  and  FI  histories.  Opinions  were  mixed  on  what  would  occur  if  you 
removed  the  AMIT  JPA  after  they  had  used  it  for  some  time,  although  most  participants  agreed 
that  the  AMIT  JPA  would  leave  a  positive  residual  effect. 

4. 1.2.4. 1  General  Impressions 

Participants  were  impressed  with  the  ease  with  which  they  learned  to  use  the  AMIT  JPA. 
Certain  features,  such  as  smooth  transitions  between  screens  and  links,  the  layout  of  tabs  to 
parallel  the  maintenance  and  troubleshooting  process,  were  singled  out  as  design  features  that 
made  the  AMIT  JPA  usable  and  user  friendly. 

4.1. 2.4.2  Access  to  TOs 

Participants  generally  agreed  that  having  access  to  the  TOs  electronically  would  save 
tremendous  time  as  opposed  to  walking  to  and  from  the  support  section  to  obtain  TOs. 
Furthermore,  technicians  were  impressed  by  the  ease  of  navigation  afforded  by  the  electronic  TO 
system  utilized  by  the  AMIT  JPA. 
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4.1. 2.4.3  Collaboration 


Participants  liked  the  collaboration  capability,  but  expressed  curiosity  with  regard  to 
identifying  available  collaborators.  Participants  also  questioned  who,  specifically,  would  give 
them  the  authority  to  talk  to  the  aircrew  or  other  personnel  (i.e..  Pro-super,  expediter,  supply, 
AGE,  etc.).  As  with  TO  usage,  participants  indicated  the  AMIT  JPA  would  save  time  by 
eliminating  the  need  to  secure  the  worksite  and  travel  to  meet  with  collaborators. 

4.1. 2.4.4  Research 

Participants  voiced  praise  for  the  ability  to  easily  and  quickly  extract  historical  data  from 
IMDS.  They  said  they  would  be  more  willing  to  do  more  complete  historical  searches  if  they 
had  the  AMIT  JPA. 

4.1. 2.4.5  Would  AMIT  Help  the  Novice? 

Each  participant  was  asked  if  he  or  she  felt  that  the  tool  would  help  Novices  become 
better  troubleshooters  more  quickly.  The  consensus  was  “Yes!”  because  of  resource  availability 
(TOs,  IMDS,  collaboration)  at  the  aircraft  and  the  time  savings  associated  with  avoiding  travel  to 
TOs,  IMDS  terminals,  and  collaborators.  Technicians  also  said  that  the  AMIT  JPA  would  help 
show  Novices  a  better  or  more  efficient  way  to  accomplish  tasks;  specifically,  referencing  the  FI 
history  capability.  Another  positive  involved  the  general  layout  of  the  graphical  user  interface 
(GUI).  The  tabs  along  the  top  of  the  interface  provided  a  useful  guide  for  Novices  in  terms  of 
following  the  correct  and  logical  path  to  repair.  That,  in  general,  would  help  keep  Novices 
focused  when  troubleshooting.  Some  Experts  felt  the  AMIT  JPA  would  help  Novices  be  more 
independent  when  troubleshooting,  thereby  freeing  Experts  to  perform  other  tasks  and  less  time 
helping  Novices  search  TOs,  IMDS,  and  other  materials. 
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4.2  Discussion 

4.2.1  “Errors”  vs.  Non-Critical  Deviations  -  Implications 
4.2.1. 1  Definitions:  Error  vs.  Non-Critical  Deviation 

Table  11  depicts  the  originally  intended  error  classification  methodology,  with  nine 
possible  outcomes.  A  tenth  outcome  was  added  before  testing,  under  the  general  classification  of 
“usability.”  Usability  errors  were  those  not  specific  to  the  presentation  or  application  of 
information,  but  rather  were  attributable  to  GUI  design  contributions  to  user  mistakes. 

Table  11  Errors  per  the  Interaction  between  System  and  User  Actions 


System 

Correct 

Incorrect 

Incomplete 

Correct 

Best  case,  the  system 
presents  correct 
information  and  users 
perform  correct  actions. 

NOTE:  Not  an  error. 

System  presents  erroneous 
information;  users  likely  choose 
one  of  two  correct  options: 

a)  Follow  incorrect  information 
as  presented  or 

b)  Realize  information  is  not 
correct  and  follow  correct 
alternative  actions 

System  does  not  present  all 
appropriate  task  information;  users 
likely  choose  one  of  two  correct 
options: 

a)  Perform  correct  actions  based 
only  on  system  presented 
information, 

b)  Realize  that  presented 
information  was  incomplete  and 
follow  correct  alternative  action 

User 

Incorrect 

System  presents  correct 
information  and  users 
absorb  all  presented 
information,  however  they 
interpret  it  incorrectly. 

Likely  Result:  Incorrect 
maintenance  action 

System  presents  erroneous 
information;  users  incorrectly 
interpret  information. 

Likely  Result:  Incorrect 
maintenance  action 

System  does  not  present  all 
appropriate  task  information;  users 
do  not  correctly  interpret  presented 
information. 

Likely  Result:  Incorrect 
maintenance  action 

Incomplete 

System  presents  correct 
information  and  users  fail 
to  absorb  all  presented 
information. 

Likely  Result:  Incorrect 
maintenance  action 

System  presents  erroneous 
information;  users  do  not  absorb 
all  presented  information. 

Likely  Result:  Incorrect 
maintenance  action 

System  does  not  present  all 
appropriate  task  information;  users 
do  not  absorb  all  presented 
information. 

Likely  Result:  Incorrect 
maintenance  action 

As  mentioned  earlier,  no  “System  Incomplete”  or  “System  Incorrect”  errors  occurred 
during  the  FDT.  All  observed  errors  fell  under  the  predefined  categories  of  “User  Incomplete” 
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and  “User  Incorrect,”  and  are  highlighted  in  Table  11.  Removing  the  system  aspect  of  the 
formula  greatly  reduced  potential  outcomes,  resulting  in  a  basic  categorization  of  user-oriented 
actions.  This  resulted  in  the  reclassification  of  errors  into  two  categories:  Critical  Errors  (those 
which  lead  directly  to  an  incorrect  repair)  and  Non-critical  Deviations.  This  distinction  is 
important,  as  it  demonstrates  AMIT’s  flexibility  in  allowing  deviations  from  a  structured  repair 
path  (Non-critical  Deviations)  while  preventing  errors  great  enough  to  critically  disrupt  the  repair 
process.  Each  error  category  and  its  implications  are  discussed  below. 

4.2.1. 2  All  Critical  Errors  in  As-ls  Condition 

Six  critical  errors  were  documented  in  the  Avionics  task  sessions,  and  eighteen  in  the 
E  &  E  sessions.  Distribution  among  experience  levels  indicated  all  Avionics  errors  were  made 
by  Experts,  while  for  the  E  &  E  tasks,  9  were  made  by  Experts,  5  by  Novices,  and  4  by  Crew 
Chiefs.  Regardless  of  experience  level,  all  Critical  Errors  occurred  in  the  As-is  condition;  these 
participants  were  using  paper  TOs.  The  majority  of  Critical  Errors  were  related  to  incorrectly 
navigating  the  TO  and  conducting  the  incorrect  ops  check,  returning  to  the  wrong  FI,  or  using  an 
inappropriate  section  of  the  TO  for  task  accomplishment  (i.e.,  using  the  FI  as  a  checklist  for  the 
ops  check).  The  remaining  Critical  Errors  involved  the  technician  relying  on  “gut  feel”  to 
ultimately  choose  an  incorrect  repair.  No  Critical  Errors  occurred  in  the  To-be  condition. 

4.2.1. 3  Non-Critical  Deviations 

Non-critical  Deviations  were  defined  as  “a  deviation  from  the  prescribed  repair  path,  but 
not  to  the  degree  the  repair  was  compromised.”  For  definition  purposes,  prescribed  meant  the 
repair  path  as  dictated  by  the  TO,  including  any  and  all  preparatory  and  close-out  steps.  Were  a 
technician  to  omit  an  operational  check  or  a  subtask  listed  in  the  technical  data,  even  if  this  step 
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was  not  necessary  to  complete  the  repair  (e.g.,  conducting  INS  alignment  in  the  complex  FCR 
repair  task),  it  was  considered  a  Non-critical  Deviation. 

While  the  TO  was  used  as  the  basis  for  defining  Non-critical  Deviations,  the  path 
prescribed  by  the  TO  may  not  have  equated  to  the  optimal,  or  most  efficient  path.  For  example, 
one  of  the  FCR  discrepancies  involved  a  failure  of  the  absolute  pressure  regulator  at  altitudes 
above  13,000  feet.  This  situation  resulted  in  the  aircraft  returning  with  a  fault  code,  which  was 
interpreted  by  many  technicians  as  legitimate.  The  technicians  may  have  considered  fault 
verification  unnecessary  because  the  discrepancy  occurred  in  the  flight  environment.  But 
because  AF  regulations  dictate  that  the  TO  be  followed  without  exception,  the  choice  to  skip 
fault  verification,  regardless  of  the  reason,  was  logged  as  a  Non-critical  Deviation.  This  did  not 
mean  the  aircraft  was  incorrectly  repaired.  Rather,  it  illustrated  the  AMIT  concept’s  flexibility  in 
allowing  technicians  the  option  to  interpret  and  follow  the  TO  according  to  level  of  expertise. 
4.2.1. 4  Information  Presentation,  Timing,  and  Errors 

As  previously  mentioned,  the  collection  and  analysis  of  errors  did  not  follow  the  original 
methodology  outlined  in  the  test  plan.  Several  factors  contributed  to  this,  but  most  significantly 
the  AMIT  JPA  did  not  contribute  to  error  occurrences.  Once  testing  began,  it  became  apparent 
that  the  AMIT  JPA  was  performing  as  intended,  and  the  errors  that  were  observed  involved  the 
user  misapplying  or  altogether  omitting  the  use  of  key  information  that  would  lead  to  the  most 
efficient  repair  path.  These  occurrences  were  flagged  as  Non-critical  Deviations  and  tracked 
across  all  experimental  conditions.  Interestingly,  the  AMIT  JPA  did  not  proportionally  reduce 
the  amount  of  Non-critical  Deviations  associated  with  the  lack  of  critical  information.  Rather,  it 
revealed,  even  with  all  critical  information,  technicians  were  equally  likely  to  commit  Non- 
critical  Deviations. 
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In  the  As-is  condition,  technicians  conducted  historical  research  on  their  own,  without 
prompting  or  encouragement.  Thus,  if  he  or  she  made  the  choice  not  to  conduct  such  an 
historical  research,  the  likelihood  of  repeating  earlier,  unsuccessful  troubleshooting  efforts  was 
high.  However,  in  the  To-be  condition,  technicians  were  always  presented  with  historical 
information  as  part  of  the  GUI  Preparation  tab.  In  either  case,  if  information  was  present,  two 
potential  types  of  Non-critical  Deviations  were  then  possible:  1)  the  information  could  be 
misapplied,  or  2)  the  information  could  be  ignored.  Misapplication  of  the  information  indicated 
the  technician  retained  the  information,  but  incorrectly  used  it.  Disuse  of  critical  information  is 
likely  rooted  in  one  of  two  causes:  1)  either  the  technician  forgot  the  information,  or  2)  the 
information  was  retained  but  conflicted  with  the  TO  and  was  thus  discarded.  The  latter  case  is 
an  AF  doctrinal  issue.  In  the  AF,  TOs  are  just  that.  Orders,  mandating  compliance  under  the 
Uniform  Code  of  Military  Justice. 

In  general,  the  FDT  results  indicate  that  automatic  access  to  information  does  equate  to 
effective  application  of  that  information.  Timing  is  the  key.  In  preparation  for  full  development 
and  field  deployment,  AMIT  designers  will  need  to  determine  a  method  to  display  relevant 
information  when  it  is  most  needed.  The  most  pertinent  example  observed  during  the  FDT  was 
the  display  of  historical  data  at  the  appropriate  steps  in  the  FI.  A  potential  way  of  accomplishing 
this  would  be  integrating  such  historical  information  as  an  official  part  of  the  technical  data.  By 
fusing  the  historical  information  with  technical  data  in  this  way,  technicians  could  see  when  a 
given  step  was  accomplished  on  a  previous  repair,  so  that  he/she  would  not  duplicate  work 
already  accomplished. 
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4.2.2  Hypotheses;  In-depth  Discussion 
4.2.2.1  Hypothesis  1:  AMIT  Will  Reduce  Time 

The  improvement  in  Task  Time  resulting  from  use  of  the  AMIT  JPA  can  be  traced  to 
several  features  not  found  in  the  current  As-is  condition.  These  features  included  focused 
historical  searches,  directed  TO  navigation,  hyperlinks  within  the  TO  framework,  and  collection 
of  repair  information.  The  role  of  historical  research,  especially  on  repeat  discrepancies,  is 
central  in  establishing  preexisting  patterns  and  tendencies  for  given  system  or  subsystem.  Such 
historical  patterns  help  guide  technicians  to  the  best  repair  when  selecting  options  from  the  FI. 
The  AMIT  JPA  provided  easy  access  and  presentation  of  this  information,  and  allowed 
technicians  to  quickly  make  decisions  on  the  appropriate  cause  of  the  discrepancy  and  helped  in 
selecting  the  subsequent  repair. 

Once  the  appropriate  repair  was  selected,  the  technician  accessed  the  TO  library  for 
instructions  on  completing  the  repair.  The  AMIT  JPA  assisted  with  this  process,  using  the  fault 
code  to  identify  the  necessary  TOs  for  the  technician  and  displaying  them  in  the  AMIT  GUI. 
Once  in  the  TO,  AMIT  again  aided  the  technician  in  navigating  to  the  appropriate  TO  chapter 
and  paragraph.  The  value  of  this  capability  was  demonstrated  by  the  absence  of  Critical  Errors 
attributed  to  TO  navigation  when  using  the  AMIT  JPA,  versus  the  significant  number  of  TO 
navigation  errors  made  in  the  As-is  condition. 

Finally,  the  AMIT  JPA  collected  information  as  the  technician  carried  out  fault 
verification,  selected  and  executed  a  repair,  and  conducted  operational  checkouts.  This 
automatic,  unobtrusive  data  collection  capability  allowed  the  AMIT  JPA  to  generate  a  logbook 
record  of  the  repair.  The  record  was  stored  in  a  standard  format  immediately  available  to  other 
AMIT  users.  Specific  comparisons  of  the  AMIT  JPA-generated  logbook  information  and  the 
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manual  logbook  entries  created  by  technician  in  the  As-is  condition  are  discussed  in  section 
4.2.5. 

Of  note  is  an  outlier  in  the  As-is  E  &  E  time  data  for  the  complex  task,  as  identified  by 
the  asterisk  in  Table  5.  This  individual,  an  Expert,  followed  the  maintenance  process  as  he 
would  have  during  an  actual  repair.  However,  this  individual’s  unorthodox  (but  legitimate) 
repair  strategy  resulted  in  an  outlying  Task  T  ime  for  the  Complex  task.  Removal  of  this 
individual  from  the  data  set  after  the  fact  would  have  created  an  “unequal  n”  situation,  but  would 
not  have  significantly  changed  the  results  of  the  statistical  analysis.  Therefore,  the  team  decided 
to  retain  this  participant’s  data,  as  the  performance  was  considered  representative  of  a  legitimate 
repair  decision  process. 

4.2.2.2  Hypothesis  2:  AMIT  Will  Reduce  Errors 

Given  that  all  24  Critical  Errors  occurred  in  the  As-is  condition.  Hypothesis  2  was  met. 
The  AMIT  JPA  did  not  contribute  to  a  single  error  that  resulted  in  an  incorrect  repair. 

For  Non-critical  Deviations,  one  might  expect  a  similar  pattern  of  results.  However,  the 
AMIT  JPA  reduced  Non-critical  Deviations  only  for  the  E  &  E  tasks,  not  the  Avionics  tasks. 
This  is  likely  attributable  to  the  difference  troubleshooting  strategies  across  the  two  specialties. 
Avionics  technicians  rely  on  Built  In  Test  (BIT)  results,  while  E  &  E  technicians  tend  to  use 
human-reported  fault  data,  which  can  then  be  verified  via  test  equipment.  In  the  FDT,  most 
Non-critical  Deviations  involved  skipping  fault  verification.  For  the  E  &  E  tasks,  the  chance  to 
verify  the  fault  came  via  a  TO  procedure,  to  which  the  AMIT  JPA  automatieally  navigated  the 
technician.  For  Avionics  tasks,  however,  a  BIT  result  had  already  “verified”  the  fault.  With  this 
information,  technicians  may  have  felt  more  comfortable  skipping  fault  verification,  regardless 
of  whether  they  were  using  the  AMIT  JPA. 
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Two  additional  factors  may  have  contributed:  technician  experience  level  and  Task 
Difficulty.  Examination  of  the  (non-significant)  interaction  of  Experience  x  Task  Difficulty  for 
the  E  &  E  tasks  showed  a  tendency  of  Experts  to  commit  more  Non-eritieal  Deviations  in  the 
complex  task,  as  compared  to  the  Simple  and  Moderate  tasks.  Conversely,  Novices  and  Crew 
Chiefs  tended  to  commit  fewer  Non-critical  Deviations  in  the  Complex  task,  as  compared  to  the 
other  tasks.  While  not  statistically  significant,  this  pattern  is  indicative  of  behavior  described  by 
the  AMUG  and  other  SMEs  —  a  tendency  for  less  experienced  technicians  to  follow  the  TO  “to 
the  letter”  in  difficult  situations.  On  the  other  hand,  more  experienced  teehnieians  may  have 
decided  to  skip  non-critical  steps  in  the  maintenance  proeess  for  the  Complex  task,  focusing 
instead  on  information  relevant  to  correct  diagnosis  and  repair.  In  either  case,  AMIT  supports 
the  teehnician’s  troubleshooting  decisions. 

4.2.2.3  Hypothesis  3:  Using  AMIT,  Novices  Can  Perform  Like  Experts 

One  of  the  stated  goals  of  the  AMIT  program  was  to  not  only  identify  and  accommodate 
the  needs  of  both  Novice  and  Expert  technicians,  but  also  move  Novice  technician  performance 
when  using  AMIT  JPA  to  a  level  similar  to  that  of  an  Expert  not  using  the  AMIT  JPA.  FDT 
performance  metrics  demonstrated  this  by  the  similarity  of  Task  Time  and  Task  Error  rates.  In 
some  cases.  Novices  using  AMIT  actually  outperformed  Experts  using  existing  processes.  The 
key  design  factor  that  produced  this  result  was  accommodating  the  needs  and  tendencies  of  the 
Novice  (e.g.,  procedural  focus  and  heavy  reliance  on  TOs),  while  also  making  available  the 
information  resourees  beyond  the  TO  that  the  Experts  use  (e.g.,  aircraft  and  maintenance 
histories,  communication  with  aircrew).  The  result  was  a  fairly  balaneed  GUI  in  terms  of  layout 
and  content,  accommodating  both  the  Novice  and  Expert  technician  t  bought  processes  and 
information  needs. 
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4.2.3  Comparing  Moderate  and  Complex  E  &  E  Task  Times 

An  interesting  quantitative  result  was  significantly  higher  task  completion  time  associated 
with  the  Moderate  E  &  E  task  as  compared  to  the  Simple  and  Complex  tasks.  It  would  seem 
logical  that  task  completion  times  would  increase  as  a  function  of  Task  Difficulty  as  they  did 
with  the  Avionics  tasks.  Elowever,  this  was  not  the  case  in  the  E  &  E  tasks,  for  two  specific 
reasons:  1)  the  cognitive  requirements  of  the  task,  and  2)  the  physical  requirements  of  the  task. 

First,  the  overall  concept  of  Task  Difficulty  was  defined  as  a  function  of  the  level  of 
cognitive  complexity  associated  with  correctly  troubleshooting,  diagnosing,  and  implementing  a 
repair.  In  the  E  &  E  task  suite,  the  Complex  task  required  the  technician  to  identify  a  solution 
that  existed  beyond  the  predefined  repairs  as  provided  by  the  TO.  To  correctly  repair  the 
aircraft,  the  technician  needed  to  deduce  from  the  results  of  a  wiring  check  that  a  relay  was 
faulty;  this  relay  was  not  identified  as  a  candidate  cause  in  the  FI. 

A  second  factor  which  contributed  directly  to  task  completion  time  was  the  time  required 
to  physically  conduct  the  repair.  The  standard  repair  time  for  replacing  the  above  mentioned 
relay  in  the  Complex  task  was  3  minutes.  The  standard  repair  time  for  the  Moderate  task  was  4 
hours.  If  these  times  are  subtracted  from  the  total  Task  Times,  the  completion  times  associated 
with  the  Complex  task  become  greater  than  the  Simple  and  Moderate  tasks  as  would  be 
expected. 

4.2.4  Collaboration:  User  Requirements  and  Available  Technology 

When  technicians  did  not  successfully  apply  historical  information  when  working  on  a 
repeat  write-up,  the  AMIT  JPA  provided  a  collaborative  capability  when  it  was  needed  most. 
However,  the  format  and  level  of  information  necessary  during  a  collaborative  session  remains 
open  for  debate. 
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As  the  collaborative  requirements  for  the  AMIT  JPA  were  developed,  members  of  the 
design  team  assumed  that  technicians  would  want  to  use  the  latest  technology  in  developing  a 
collaborative  capability.  Streeiming  video,  shared  workspace  and  high-fidelity  image 
transmission  and  audio  were  seen  as  not  only  technologically  possible,  but  necessary.  Contrary 
to  this  assumption,  review  sessions  with  the  AMUG  revealed  that  technicians  were  more  inclined 
to  seek  out  another  technician  if  the  collaborative  session  required  more  than  instant  messaging. 
Indeed,  a  review  of  the  number  of  collaborative  sessions  during  the  FDT  indicates  that 
technicians  collaborated  nearly  twice  as  often  in  the  As-is  condition  than  when  using  the  AMIT 
JPA.  This  basic  pattern  held  for  both  the  Avionics  and  E  &  E  tasks,  with  the  majority  of 
collaborative  sessions  occurring  in  the  Complex  task.  Tables  12  and  13  illustrate  this  finding, 
which  begs  the  question,  “Why,  with  an  easily  accessible  collaborative  capability,  did  AMIT 
users  collaborate  less  than  non-AMIT  users?" 

The  Team  believes  the  AMIT  JPA  provided  enough  relevant  and  timely  information  to 
minimize  the  need  to  collaborate.  By  design,  the  AMIT  JPA  provided  answers  to  certain 
anticipated  questions  that  might  warrant  collaboration;  for  example,  the  next  repair  step  as  per 
the  FI.  The  AMIT  JPA  negated  the  need  to  ask  many  questions  by  playing  the  role  of  “surrogate 
SME.”  That  is,  in  all  situations  save  the  Complex  FCR  repair  task,  the  AMIT  JPA  provided 
information  immediately  to  a  technician  who  may  otherwise  have  dedicated  time  and  resources 
to  find  and  query  a  SME.  Thus,  AMIT  not  only  saved  time,  but  also  kept  available  an  important 
information  resource  (the  SME)  for  potentially  more  difficult  situations  that  would  benefit  from 
his  or  her  expertise. 
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Table  12  Collaboration  Totals  for  all  Task  Difficulty  Levels,  Avionics 


As-is 

To-be 

Simple 

Moderate 

Complex 

Simple 

Moderate 

Complex 

Novice 

6 

3 

13 

1 

1 

12 

Expert 

4 

5 

9 

3 

0 

5 

Crew  Chief 

5 

3 

9 

0 

0 

12 

Table  13  Collaboration  Totals  for  all  Task  Difficulty  Levels,  E  &  E 


As-is 

To-be 

Simple 

Moderate 

Complex 

Simple 

Moderate 

Complex 

Novice 

1 

4 

4 

0 

0 

4 

Expert 

1 

1 

6 

0 

0 

2 

Crew  Chief 

3 

1 

12 

0 

2 

7 

Notable  parallels  were  observed  in  the  patterns  of  collaborative  behavior  between  the 
Avionics  and  E  &  E  tasks.  Experts  generally  did  not  collaborate  with  other  Experts  in  the  As-is 
or  To-be  conditions.  In  terms  of  the  effect  of  Task  Difficulty  on  the  amount  of  collaboration,  the 
majority  of  all  collaboration  was  accomplished  on  Complex  problems.  However,  noticeable 
differences  in  collaboration  frequency  were  observed  during  the  Simple  and  Moderate  tasks. 
Finally,  in  the  Avionics  task,  which  involved  an  in-flight  discrepancy  that  could  not  be 
duplicated  on  the  ground.  Experts  collaborated  with  pilots  more  often  than  in  other  cases. 

An  additional  issue  involves  the  technician’s  actual  level  of  collaborative  need.  AMUGs 
repeatedly  stated  once  the  need  to  collaborate  went  beyond  the  level  of  an  instant  message  (IM), 
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a  technician  would  probably  seek  out  a  collaborator  in  person.  Design  Consideration  Testing 
helped  confirm  the  practicality  of  IM;  as  such,  it  was  chosen  as  the  AMIT  JPA  collaborative 
capability.  However,  the  level  of  information  exchange  described  by  the  AMUG  (i.e.,  the  “in- 
person”  meeting  with  the  collaborator)  is  achievable  with  existing  technology.  Specifically, 
collaborative  technologies  such  as  “shared  workspace”  can  give  technicians  and  remotely  located 
collaborators  literal  “same  page”  synchronicity  during  a  collaborative  session.  These  capabilities 
were  discussed  during  the  AMUG  sessions,  but  were  not  demonstrated  or  evaluated.  While  all 
AMUG  members  were  familiar  with  IM,  the  concept  of  a  shared  workspace  capability  may  not 
have  been  fully  understood,  and  therefore  overlooked. 

4.2.5  AMIT  JPA  Effect  on  Logbook  Entries 

The  role  of  the  maintenance  logbook  has  many  functions,  including  a  general  historical 
record,  a  place  to  add  information  that  IMDS  does  not  ordinarily  collect,  and  a  means  for 
communicating  unusual  or  non-standard  conditions  and  repairs  associated  with  a  particular 
aircraft.  While  such  information  is  considered  invaluable  within  a  squadron,  it  is  rarely  shared 
with  other  organizations.  Furthermore,  no  AF  standard  exists  in  terms  of  configuring  a  logbook, 
or  even  what  information  or  the  appropriate  level  of  detail  should  be  included  in  a  given  entry. 

The  test  team  did  not  develop  metrics  to  directly  compare  information  collected  by  the 
AMIT  JPA  with  the  simulated  e-logbook.  However,  technicians  indicated  the  AMIT  JPA 
logbook  feature  would  add  value  to  the  overall  collection  of  maintenance  information.  AMIT 
SMEs  recommended  that  any  logbook  entry  should  include  the  aircraft  tail  number,  description 
of  the  discrepancy,  and  the  corrective  action.  Examination  of  logbook  entries  made  by 
technicians  in  the  As-is  condition  illustrates  how  the  AMIT  logbook  capability  would  help  to 
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increase  input  accuracy,  standardize  format  for  re-use,  and  help  maintainers  avoid  critical 
omissions  during  log  entries. 

A  total  of  130  log  entries  were  created  by  36  different  technicians  in  the  As-is  condition, 
using  a  representative  e-logbook  format  as  provided  by  the  63'^'^  Aircraft  Maintenance  Unit,  756* 
Aircraft  Maintenance  Squadron  at  Luke  AFB.  A  review  of  logbook  entries  made  in  the  As-is 
condition  revealed  a  series  of  input  errors.  Seven  technicians  entered  incorreet  tail  numbers 
when  entering  data  into  the  log.  Further,  28  entries  were  made  without  noting  the  discrepancy. 
The  AMIT  JPA  logbook  function  addresses  such  issues  by  automatically  collecting  key 
information  and  decisions  made  during  the  maintenance  process,  thus  relieving  the  technieian  of 
data  collection  burdens  at  the  end  of  his  or  her  shift. 

4.2.6  AMIT  Facilitates  Efficient  Use  of  Time 

The  AMIT  JPA  facilitated  a  more  efficient  use  of  time.  This  was  illustrated  by  the 
capability  to  conduct  historical  searches  before  going  to  the  aircraft  and  continued  as  the  AMIT 
JPA  facilitated  step-by-step  procedural  guidance  through  the  tech  data,  including  steps  that  might 
be  skipped  when  using  paper  TOs  (e.g.,  fault  verification).  Conversely,  technicians  in  the  As-is 
condition  spent  more  time  searching  the  TO  library,  rather  than  historical  maintenance  data. 
These  technicians  were  necessarily  searching  through  paper-based  technical  data  to  find  the 
appropriate  procedures  for  the  given  discrepancy. 

Deeper  evaluation  of  Task  Time  data  highlighted  technicians  using  the  AMIT  JPA  spent 
a  greater  amount  of  time  conducting  research  and  fault  verification  tasks,  yet  total  Task  Times 
for  those  using  the  AMIT  JPA  were  lower  than  those  not  using  the  JPA.  While  statistical 
analyses  were  not  performed  on  this  data,  this  pattern  of  results  illustrates  a  key  benefit  of  the 
AMIT  JPA. 
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A  counterintuitive  finding  was  technicians  using  the  AMIT  JPA  spent  less  time 
collaborating  than  those  in  the  As-is  condition.  Two  potential  reasons  exist  for  this  observation. 
First,  as  noted  earlier,  technicians  found  the  information  they  needed  in  the  To-be  condition 
without  consulting  with  additional  personnel.  Second  (and  not  as  positive)  technicians  may  have 
found  the  AMIT  collaboration  capability  cumbersome,  as  their  only  option  was  to  type  an  instant 
message.  Collaboration  results  notwithstanding,  the  reallocation  of  time  in  the  To-be  condition 
illustrates  a  more  efficient  focus  on  the  maintenance  process  in  terms  of  both  resource  allocation 
and  task  completion. 

4.2.7  Interpretation  of  the  Subjective  Results 

While  collection  of  subjective  feedback  was  not  intended  to  undergo  statistical  analysis,  a 
number  of  inferences  can  be  dravra  from  the  subjective  data.  Nearly  every  participant,  whether 
they  used  the  AMIT  JPA  or  saw  the  demonstration  afterwards,  asked,  “When  will  the  AMIT  JPA 
be  implemented?”  The  Team  perceived  this  as  a  firm  validation  of  the  AMIT  concept  and  an 
endorsement  that  the  JPA  is  needed  and  wanted  by  those  for  whom  it  was  designed.  The  AMIT 
JPA’s  design  for  seamless  integration  with  legacy  AF  systems  prepares  the  way  for  quick, 
efficient  integration  into  existing  and  emerging  AF  logistics  infrastructure.  A  second,  well- 
received  global  benefit  was  the  notion  that  the  AMIT  JPA  could  not  only  use  existing  historical 
data,  but  also  build  a  more  detailed  historical  database  for  re-use.  In  general,  technicians  were 
impressed  that  the  AMIT  JPA  was  designed  to  work  with  existing  AF  systems,  as  opposed  to 
another  stand  alone  system  requiring  protracted  installation,  training,  and  debugging  time. 

Subjective  feedback  on  specific  design  characteristics  and  capabilities  were  also 
generally  positive.  Technicians  liked  the  overall  design  of  the  user  interface,  and  remarked  that 
its  layout  would  benefit  both  Novice  and  Expert  technicians.  Directed  TO  searches;  that  is,  the 
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ability  of  AMIT  to  navigate  automatically  to  the  section  of  the  TO  library  appropriate  for  the 
given  fault  code,  was  often  favorably  noted.  This  capability  was  specifically  mentioned  by 
Expert  technicians  as  a  feature  that  would  help  Novices  learn  more  quickly,  making  the 
connection  between  specific  faults  and  specific  areas  of  the  TO  library.  Focused  historical 
search  capability  and  the  automatic  collection  of  logbook  data  also  garnered  praise  for  most 
technicians.  Both  of  these  capabilities  were  seen  as  time  savers,  especially  the  logbook 
capability,  which  is  sometimes  seen  as  a  “necessary  evil”  along  with  IMDS  data  entry.  In  fact, 
several  technicians  spoke  of  being  directed  to  stop  work  on  the  aircraft  specifically  to  complete 
the  required  IMDS  documentation  per  AF  protocol,  hindering  overall  repair  efforts. 

The  only  AMIT  capability  that  received  a  lukewarm  response  was  the  collaborative 
capability.  Overall,  technicians  were  skeptical  that  an  aircrew  would  be  available  for  direct 
question  and  answer  sessions,  as  was  simulated  in  the  FDT  scenarios.  In  fact,  technicians  felt  the 
primary  benefit  of  such  a  capability  was  the  ability  to  contact  other  maintenance  personnel  from 
the  point  of  maintenance,  and  said  it  would  most  likely  be  used  to  request  the  presence  of  the 
Production  Superintendent  or  another  technician.  Such  feedback  recalls  the  question  of  just  how 
robust  the  design  of  a  collaboration  capability  should  be,  as  discussed  above.  In  general, 
technicians  felt  that  a  collaborative  capability  should  at  least  reflect  current  practices  by 
including  audio,  which  the  AMIT  JPA  did  not.  Other  potential  capabilities  include  a  shared 
workspace  and  the  ability  to  email  a  remotely  located  SME,  who  may  not  be  at  the  ready  for  real 
time  information  exchange.  In  general,  technicians  felt  that  collaboration,  at  least  in  the  form  of 
instant  messaging,  would  provide  only  minimal  assistance. 
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4.3  Cost  Analysis  (CA) 

4.3.1  Overview 

The  AMIT  cost  analysis  (CA)  purposes  to  quantify  time  (personnel  and  aircraft 
availability)  and  dollar  savings  potential  based  on  AMIT  FDT  results.  To  accomplish  this,  the 
AMIT  CA  acknowledges  three  data  sources:  1)  FDT  results,  2)  Air  Force  Data  Services 
(AFDS),  a  repository  for  historical  maintenance  data,  and  3)  the  President’s  2006  defense 
budget.  In  concert,  these  three  sources  provide  projected  time  savings  data,  a  baseline  accoimt  of 
actual  maintenance  task  times  on  numerous  AF  aircraft,  and  technician  (labor)  costs.  These 
sources  facilitate  calculations  of  cost  associated  with  the  F-16C  Block  40/42,  but  also  afford  the 
opportunity  to  extrapolate  to  other  AF  weapons  systems.  Because  inferential  assumptions  must 
be  made  regarding  the  applicability  of  both  the  FDT  data  and  AFDS,  maintenance  data  to  other 
weapons  systems  and  certain  of  trade-offs  exist.  Figure  12  depicts  the  nature  of  these  trade-offs. 
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The  following  sections  detail  the  methodology  for  calculating  costs  associated  with  the 
F-16C  Block  40/42  weapon  system  and  projecting  savings  due  to  AMIT  implementation.  A 

detailed  description  of  the  AMIT  CA,  including  extrapolation  to  additional  F-16  blocks  and 
other  aircraft,  is  in  Appendix  A. 

4.3.2  F-16C  Block  40/42  CA  Results 

The  AMIT  FDT  collected  performance  data  with  the  intent  of  illustrating  AMIT  impacts 
on  the  aircraft  maintenance  community.  The  FDT  data  used  in  this  CA  only  includes  the  average 
task  time  collected  for  the  system  specific  expert  and  novice  user  groups.  The  average  task  time 
collected  for  the  Crew  Chief  user  group  was  excluded  here  because  Crew  Chiefs  would  not 
normally  perform  troubleshooting  tasks  for  Avionics  and  E  &  E  systems.  Using  FDT  results, 
actual  AFDS  maintenance  data,  and  cost  data  from  the  President’s  2006  budget,  the  AMIT  CA 
indicates: 
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Time  savings  of  45  minutes  per  task  for  the  Avionics  tasks 
Time  savings  55  minutes  per  task  for  the  E  &  E  tasks 
The  second  variable  in  calculating  an  AMIT  cost-benefit  was  representative  historical  repair 
data.  Five  months  of  F-16  maintenance  data  was  drawn  from  the  AFDS  database,  examined,  and 
approved  by  SMEs  in  terms  of  overall  level  of  representation. 

Not  every  maintenance  action  involves  troubleshooting;  those  that  do  will  realize  the 
greatest  benefit  from  AMIT.  Thus,  a  key  calculation  in  the  AMIT  CA  was  the  proportion  of 
maintenance  actions  from  the  AFDS  data  that  required  troubleshooting.  Detailed  analysis  by 
AMIT  SMEs  indicated  that  at  least  50%  of  the  total  actions  involve  troubleshooting.  Prorated 
over  12  months,  these  figures  indicate  approximately: 

16,742  Avionics  tasks  per  year  involve  troubleshooting 
37,747  E  &  E  tasks  per  year  involve  troubleshooting 
Application  of  the  FDT  time  savings  figures  yields: 

annual  time  savings  of  12,557  clock-hours  for  Avionics  repairs 
annual  time  savings  of  34,601  clock-hours  for  E  &  E  repairs 
combined  increase  in  aircraft  availability  of  47,158  clock-hours 
In  order  to  project  the  impact  of  this  savings  at  the  fleet  level,  47,158  availability  hours  were 
divided  by  8,760  hours  per  year  (365  x  24),  resulting  in: 

an  equivalent  increase  of  5.4  F-16C  Block  40/42  aircraft  per  year 
In  order  to  convert  these  figures  to  man-hour  savings,  the  clock-hour  savings  noted  above  were 
multiplied  by  the  average  crew  size  of  two,  resulting  in: 
savings  of  94,316  man-hours  per  year. 
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Man-hour  cost  was  calculated  by  multiplying  the  man-hour  savings  by  the  average  hourly 
(burdened)  cost  of  maintenance  technicians  as  stated  in  the  2006  President’s  defense  budget 
(API  65-503,  Table  A20-1).  This  average  hourly  wage  for  E-4,  E-5,  and  E-6  technicians  is 
$30.03.  Therefore;  an  annual  labor  dollar  savings  of  $2,828,561  can  be  realized  for  the  F-16C 
Block  40/42  fleet  for  Avionics  and  E  &  E  tasks  alone. 

Note  this  dollar  figure  represents  labor  hours  only  and  does  not  include  other  cost 
considerations,  such  as  unnecessary  use  of  parts.  Additionally,  these  figures  do  not  highlight  the 
intangible  benefits  realized  in  personnel  availability.  Novice  performance  enhancements, 
scheduling  impacts,  and  training  impacts. 

4.4  Summary 

4.4.1  Why  Did  AMIT  Work? 

As  a  research  project  conceived  with  the  goal  of  proving  a  concept,  AMIT  succeeded. 
Beyond  this  goal,  however,  the  team  designed  and  tested  a  prototype  solution  for  the 
maintenance  community  that  has  been  embraced  by  technicians,  managers,  and  senior  leaders 
alike.  While  the  AMIT  concept  of  bringing  together  disparate  information  sources  to  the 
technician  is  not  necessarily  cutting-edge,  the  overwhelming  positive  response  from  the 
maintenance  community  was  the  result  of  a  methodical  process.  That  is,  inclusion  of  the  user 
community  from  the  project’s  outset  and  a  visionary  perspective  toward  integration  with  existing 
and  anticipated  AF  infrastructure.  In  the  end,  the  AMIT  concept’s  potential  for  success  and 
implementation  lies  in  the  background  and  evaluation  work  accomplished  throughout  the  3  years 
of  the  program. 

The  first  year  of  the  AMIT  project  was  dedicated  to  understanding,  evaluating,  and 
prioritizing  user  requirements.  With  an  understanding  that  Novice  and  Expert  technicians  do 
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things  differently,  the  team  was  aware  from  the  outset  that  they  would  he  designing  for  at  least 
two  groups  of  users.  Thus,  year  one  was  dedicated  to  detailed  analysis  of  maintenance  processes 
from  Novice  and  Expert  perspectives,  and  subsequently  generating  a  series  of  requirements  that 
would  benefit  both  groups  of  users.  This  was  accomplished  with  a  series  of  Process  Interviews, 
Cognitive  Task  Analyses,  and  a  Literature  Review.  The  combined  results  of  these  three  efforts 
formed  the  basis  for  the  early  GUI  design. 

Over  the  next  year  of  the  project,  GUI  development  continued.  During  this  period,  the 
user  interface  was  shown  to  the  potential  end-user  community  via  a  series  of  AMUG  sessions. 
The  goal  of  these  sessions  was  to  get  “reality  checks”  from  the  end-user  community  regarding 
the  overall  design  of  the  GUI  to  the  prioritization  of  certain  capabilities,  as  well  as  terminology 
usage.  The  results  of  the  AMUG  sessions  helped  to  solidify  much  of  the  final  design  of  the 
AMIT  JPA,  both  in  terms  of  appearance  and  capability. 

A  series  of  Design  Consideration  Tests  (DCTs)  were  conducted  to  address  a  short  list  of 
design  features  that  had  not  been  finalized  via  end-user  feedback.  The  DCTs  were  “mini  field 
studies,”  and  as  with  the  AMUG  sessions,  using  aircraft  maintenance  technicians  as  participants. 
Results  of  the  DCTs  answered  the  final  array  of  design  questions,  including  prompting  (icon  vs. 
dialogue  box  vs.  screen  overlay),  collaboration  (voice  vs.  IM  vs.  IM  with  shared  graphical 
workspace),  data  entry  technique  (voice  vs.  keyboard),  data  retrieval  format  (audio  vs.  text),  and 
enhanced  electronic  logbook  format  (inclusion  of  “breadcrumb  trails”  and  integrated  historical 
information). 

The  final  AMIT  JPA,  validated  by  the  results  of  the  FDT  in  year  three,  represents  a 
design  that  from  the  beginning  has  incorporated  the  end-user  community  as  an  evaluative 
mechanism.  The  resultant  tool  brings  information  resources  together,  focuses  relevant 
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information  to  the  user  in  a  format  that  he/she  can  understand,  while  reducing  the  physical  and 
cognitive  overhead  associated  with  gathering  and  sorting  through  the  TO  library.  The  automatic 
collection  of  key  information  throughout  the  maintenance  process  relieves  the  technician  of  the 
need  to  mentally  review  and  document  the  repair  at  the  end  of  the  shift.  Finally,  the  information 
is  presented  in  a  configuration  that  is  understandable  by  all  members  of  the  AF  maintenance 
community. 

4.4.2  Why  Will  AMIT  Work? 

From  an  implementation  perspective,  the  AMIT  JPA’s  strength  lies  in  its  design  to  work 
with  existing  AF  systems,  while  creating  a  new,  robust  data  set  that  is  immediately  available  for 
reuse  during  later  troubleshooting  activities.  Throughout  the  project’s  duration,  briefings  to 
upper  level  management  personnel  resulted  in  praise  and  positive  comments.  As  word  of  the 
AMIT  project  spread  throughout  the  AF,  unsolicited  positive  feedback  from  other  groups  was 
received  as  well.  Added  to  the  resounding  endorsement  from  the  technician  community  itself,  all 
levels  of  the  AF  maintenance  community  clearly  express  a  desire  to  see  AMIT  transition  into  a 
fielded  system. 
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5  Conclusion 

The  AMIT  concept,  instantiated  by  the  AMIT  JPA,  proved  beneficial  to  troubleshooters 
in  the  operational  environment  at  Luke  AFB,  Arizona.  The  Field  Demonstration  Test  (FDT) 
results  clearly  point  to’ward  an  easily  implemented,  cost  effective  capability  with  high  potential 
to  meet  the  goals  of  AF  leadership,  including  improved  aircraft  availability  (e.g.,  readiness), 
reduced  cost,  and  reduced  deployment  footprint.  As  highlighted  in  the  FDT,  technician 
performance  can  be  dramatically  improved,  thereby  positively  affecting  Task  Time,  Task 
Accuracy,  and  technician  proficiency.  The  demonstrated  savings  of  45  to  55  minutes  per  repair 
translates  to  over  47,000  clock  hours  saved,  per  year,  on  the  fleet  of  F-16  Block  40/42  aircraft, 
equating  to  an  approximate  increase  of  5.4  available  aircraft.  Additionally,  man-hour  dollar 
savings  of  $2,828,561  per  year  are  plausible  for  the  F-16C  Block  40/42  fleet  alone.  The  CA  in 
Appendix  A  shows  implementation  across  additional  aircraft  fleets  will  produce  similar  savings 
potential. 

As  shown  in  the  cost  analysis  (paragraph  4.3  and  Appendix  A),  improvements  in  these 
metrics  can  only  positively  impact  aircraft  Non-Mission  Capable  rates,  8-hour  fix  rates, 
persoimel  availability,  reduced  supply  demand,  and  deployment  footprint. 

Another  noteworthy  program  conclusion  is  technician  acceptance  of  the  AMIT  JPA,  and 
desire  for  near-term  implementation.  Nearly  all  test  subjects  indicated  an  immediate  need  for 
such  a  performance  aid  and  asked  when  they  might  see  AMIT  JPAs  on  their  flightline.  Almost 
every  participating  Expert  technician  agreed  that  AMIT  would  certainly  improve  on-the-job 
training,  skill  progression,  and  overall  competence  across  any  aircraft  maintenance  career  field. 
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6  Recommendations 

6.1  Transition 

Based  on  the  results  of  the  FDT  and  the  potential  savings  illuminated  by  the  CA,  the 
AMIT  Team  recommends  immediate  transition  of  the  AMIT  concept  and  JPA  to  an  operational 
command,  preferably  Air  Combat  Command  (ACC),  for  pilot  implementation.  This  transition  is 
necessary  to  move  the  AMIT  concept  from  an  advanced  research  and  development  initiative  to  a 
real-world  pilot  application.  The  proposed  pilot  is  not  intended  to  be  a  production-ready  system. 
Rather  it  will  build  upon  the  tested  and  proven  AMIT  JPA  by  establishing  technological 
integration  points  and  relevant  interfaces  within  the  GCSS-AF  Integration  Framework,  the  Air 
Force  Data  Services  (AFDS),  the  Air  Force  Portal,  and  the  technical  order  services  module  in 
GCSS-AF.  These  activities  will  pave  the  way  for  optional  MAJCOM  and/or  Air  Force-wide 
implementation. 

AMIT  transition  must  occur  immediately  upon  program  completion  to  facilitate 
development  of  the  pilot  application  in  calendar  year  2007.  A  successful  AMIT  pilot  in  2007 
would  bridge  the  advanced  research  effort,  which  ends  in  January  2007,  with  the  Program 
Objective  Memorandum  (POM)  established  by  the  ACC  for  its  Combat  Air  Forces  (CAF)  needs 
in  2008.  Appendix  B  contains  detailed  AMIT  transition  activities  and  recommendations. 

6.2  Future  Research 

6.2.1  Explore  Fused  Historical  and  FI  Views 

The  collection,  use,  and  application  of  historical  data  are  key  AMIT  features.  The 
capability  of  the  AMIT  JPA  to  provide  relevant  historical  repair  information,  especially  in  the 
case  of  repeats  and  recurs,  reduces  duplication  of  effort  and  saves  time,  ultimately  reducing 
maintenance  costs.  Simply  providing  this  information  at  the  beginning  of  the  maintenance 
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process  (e.g.,  within  the  Discrepancy  tab)  as  done  during  the  FDT,  however,  does  not  facilitate  a 
full  understanding  of  exactly  how  and/or  when  to  apply  the  information.  Historical  data 
generally  is  most  applicable  during  troubleshooting  when  the  maintainer  is  likely  to  repeat  tasks 
that  were  previously  accomplished,  wasting  time  and  adding  no  value. 

The  recommended  solution  is  to  display  relevant  historical  data  as  an  inherent  component 
of  the  FI  procedure.  Future  research  should  address  when,  where,  and  how  historical  information 
should  be  fused  with  fault  isolation  procedures.  Results  of  such  a  research  effort  will  directly 
contribute  to  reaching  AF  aircraft  availability  goals  by  enhancing  troubleshooter  effectiveness. 
Primary  considerations  during  the  research  should  include  the  amoimt  of  information  needed  for 
an  effective  fusion  strategy,  whether  it  is  possible  to  include  too  much  information,  and  whether 
fusing  historical  and  FI  information  facilitates  a  change  in  general  repair  strategy  at  all.  Answers 
to  these  questions  will  help  drive  the  next  version  of  AMIT’s  historical  and  electronic  TO 
capability,  as  well  as  address  cultural  issues  involving  maintenance  TO  format  and  usage. 

6.2.2  Increase  Collaborative  Capabilities  in  Future  Versions 

During  the  design  and  development  process,  the  team  collected  and  evaluated  basic 
collaborative  requirements.  Feedback  from  the  AMUG  indicated  a  fairly  basic,  "conversational" 
level  of  need,  beyond  which  the  technician  would  seek  out  an  SME  in  person.  While  the  AMUG 
discussed  additional  collaborative  capabilities  such  as  shared  workspace,  streaming  video,  and 
real  time  audio,  no  demonstrations  were  seen  first-hand.  Application  of  these  capabilities  may 
not  have  been  fully  understood,  and  their  inclusion  was  not  highly  recommended  for  the  FDT. 
All  AMUG  members  indicated  a  familiarity  with  IM;  however  it  was  subsequently  developed  as 
the  AMIT  JPA  collaborative  capability. 
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While  the  AMUG  did  not  indicate  a  need  for  anything  beyond  IM,  observations  from  the 
FDT  revealed  that  the  majority  of  collaborative  sessions  (in  both  the  As-is  and  To-be  conditions) 
involved  some  reference  to  the  TOs.  For  example,  most  repeat/recur  scenarios  involved 
collaboration  with  the  SME  to  review  options  on  the  FI.  During  such  sessions,  a  shared 
workspace  capability  would  allow  the  technician  and  SME  to  literally  be  “on  the  same  page”  and 
potentially  save  time.  Other  situations  may  be  even  more  complex,  requiring  an  exchange  of 
information  that  is  most  easily  evaluated  via  a  visual  or  aural  inspection.  Aircraft  battle  damage, 
for  example,  might  involve  both.  A  more  robust  collaborative  capability  would  allow  the 
technician  to  utilize  these  additional  modes  of  communication,  rather  than  generating  a  text 
description  for  the  SME.  Future  versions  of  the  AMIT  JPA  should  include  such  additional 
collaborative  features,  at  least  in  an  evaluative  form,  to  demonstrate  further  savings  in  time  and 
efficiency. 
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9  List  of  Symbols,  Abbreviations,  and  Acronyms 

List  of  Acronyms 


Acronym 


Meaning 


Air  Combat  Command 


Air  Education  Training  Command 


Air  Force 


Air  Force  Data  Services 


Air  Force  Research  Laboratory,  Logistics  Readiness  Branch,  Fiuman  Effectiveness  Direetorate 


Air  Force  Specialty  Code 


Air  Force  Smart  Operations  for  the  21''  Century 


Air  Force  Special  Operations  Command 


Aircraft  Ground  Equipment 


Air  Mobility  Command 


Aircraft  Maintenance  Intuitive  Troubleshooting 


Aircraft  Maintenance  Users  Group 


Analysis  of  Variance 


Operational  Availability 


Action  Taken 


Built-in  Test/Self  Test 


Cost  Analysis 


Combat  Air  Force 


Cost  Benefit  Analysis 


Contract  Data  Requirement  List 


Central  Command 


Combat  Information  Transportation  System 


Can  Not  Duplicate 


Combatant  Command 


Commercial-off-the-shelf 


Cognitive  Task  Analysis 


Design  Consideration  Test 


Degree  of  Freedom 


Degree  of  Freedom,  Error 


Degree  of  Freedom,  Variable 


Department  of  Defense  Instruction 


Acronym 

Meaning 

eLOG21 

Expeditionary  Logistics  for  the  2 1  ®  Century 

EUCOM 

European  Command 

FCR 

Fire  Control  Radar 

FCS 

Flight  Control  System 

FDT 

Field  Demonstration  Test 

FI 

Fault  Isolation 

FY 

Fiscal  Year 

GCSS-AF 

Global  Combat  Support  System  -  Air  Force 

GFE 

Government  Furnished  Equipment 

GWOT 

Global  War  on  Terrorism 

HAF 

Headquarters  Air  Force 

IM 

Instant  Messaging 

IMDS 

Integrated  Maintenance  Data  System 

IT 

Information  Technology 

JFCOM 

Joint  Forces  Command 

JPA 

Job  Performance  Aid 

Lit 

Literature 

LOA 

Logistics  Officer  Association 

M 

Million 

MAF 

Mobility  Air  Forces 

MAJCOM 

Major  Command 

NATO 

North  Atlantic  Treaty  Organization 

NCI 

NCI  Information  Systems,  Inc. 

OhioLINK 

Ohio  Library  and  Information  Netwoik 

Ops 

Operations 

PACOM 

Pacific  Command 

PERSTEMPO 

Personnel  Tempo 

POM 

Program  Objective  Memorandum 

R&D 

Research  and  Development 

RALT 

Radar  Altimeter 

RE  21 

Repair  Enterprise  for  21“'  Century 

SME 

Subject  Matter  Expert 

TF  720 

Task  Force  720 

TO 

Technical  Order 

TRANSCOM 

United  States  Transportation  Command 

TTP 

Technology  Transition  Plan 

UCMJ 

Uniform  Code  of  Military  Justice 

UDRI 

University  of  Dayton  Research  InstitUe 

WS 

Weapon  System 
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Appendix  A:  Cost  Analysis  (CA) 

A.1.  Overview 

In  general,  a  Cost  Benefit  Analysis  (CBA)  compares  monetary  calculations  of  initial  and 
ongoing  expenses  against  expected  return.  Constructing  plausible  measures  of  the  costs  and 
benefits  of  specific  actions,  however,  is  often  very  difficult  (Frank,  R.H.  2000).  In  practice,  an 
estimation  of  costs  and  benefits  is  accomplished  via  survey  methods  or  by  drawing  inferences 
from  general  behavior.  The  AMIT  cost  analysis  (CA)  sought  to  quantify  time  (personnel  and 
aircraft  availability)  and  dollar  savings  potential  based  on  AMIT  field  demonstration  test  (FDT) 
results.  To  accomplish  this,  the  AMIT  CA  acknowledged  three  data  sources:  1)  FDT  results,  2) 
Air  Force  Data  Services  (AFDS),  a  repository  for  historical  maintenance  data,  and  3)  the 
President’s  2006  defense  budget.  In  concert,  these  three  sources  provided  projected  time  savings 
data,  a  baseline  account  of  actual  maintenance  task  times  on  numerous  AF  aircraft,  and 
technician  (labor)  costs.  These  sources  facilitated  calculations  of  cost  associated  with  the  F-16 
Block  40/42,  but  also  afforded  the  opportunity  to  extrapolate  to  other  Air  Force  (AF)  weapon 
systems.  Because  inferential  assumptions  must  be  made  regarding  the  applicability  of  both  the 
FDT  data  and  AFDS  maintenance  data  to  other  weapons  systems,  certain  of  trade-offs  exist. 
Figure  A-1  depicts  the  nature  of  these  trade-offs.  When  extrapolating  beyond  the  FDT  aircraft, 
confidence  in  the  CA  decreases. 
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Figure  A-1  Extrapolation  Risk  Increases  when  Results  are  applied  Beyond  the  Test  Environment 


The  extensibility  of  the  AMIT  CA  depends  on  a  specific  set  of  assumptions.  Each 
assumption  builds  on  the  previous,  and  as  A-1  illustrates,  a  reduction  of  confidence  as 
extrapolation  moves  from  the  test  environment  airframe  (F-16C  Block  40/42)  to  the  overall  AF 
aircraft  fleet.  The  assumptions,  listed  for  the  applicable  specialties  of  Avionics  and  E  &  E,  are  as 
follows: 

la.  Avionics  savings  observed  during  the  AMIT  FDT  can  be  applied  across  the  entire 
Avionics  suite  for  F-16C  Block  40/42. 

lb.  E  &  E  savings  observed  during  the  AMIT  FDT  can  be  applied  across  the  entire 
E  &  E  suite  for  F-16C  Block  40/42. 

2.  These  savings  are  representative  of  savings  that  can  be  applied  to  all  functional 
systems  on  the  F-16C  Block  40/42. 
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3. 


Savings  are  representative  of  those  for  all  systems  and  blocks  of  F-16  aircraft. 


4.  Savings  are  representative  of  those  for  all  systems  on  other  weapon  systems  (C- 
130,  A- 10,  F-15,  etc.) 

A.2.  Methodology  and  Calculations 

A  basic  requirement  of  any  CA  is  the  notion  of  data  integrity.  The  AMIT  CA  data  was 
drawn  from  AFDS.  The  initial  data  set  was  pulled  based  on  Action  Taken  (AT)  codes  to  identify 
documented  maintenance  actions  most  likely  to  require  troubleshooting.  The  initial  set  contained 
5  months  of  F-16  Block  40/42  maintenance  data  and  covered  all  systems.  This  set  contained 
43,142  records  of  interest.  These  records  were  reviewed  by  AMIT  SME  team  to  ensure 
relevance.  For  example,  an  unusually  high  or  low  number  of  maintenance  actions  would  raise 
concerns  as  to  whether  the  data  represented  “typical”  maintenance  activity  for  the  time  period  in 
question. 

This  initial  set  excluded  all  actions  associated  with  daily  inspections,  cannibalization, 
scheduled  maintenance,  loading,  arming,  de-arming,  and  all  off-equipment  maintenance. 
Therefore,  the  following  on-equipment  AT  codes  were  used: 
o  R  -  Remove  and  Replace 

o  P  -  Removed  with  an  assumed  follow-on  action  of  Q  -  Installed 
o  G  -  Repair  and/or  replacement  of  minor  parts 

o  L  -  Adjust  to  include  adjustments  necessary  for  proper  functioning  of  equipment 
o  Y  -  Troubleshoot 

The  SME  team  then  calculated  a  one-year  total  of  103,541  relevant  maintenance  actions 
in  the  initial  data  set  for  the  F-16  Block  40/42  aircraft,  based  on  the  5-month  data  set  (43,172  /  5 

X  12). 
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The  Team  associated  the  work  unit  code  (WUC)  with  aircraft  system/sub-systems  to 
identify  the  subset  of  data  relevant  to  the  Avionics  and  E  &  E  suites,  allowing  FDT  result 
application  across  both  specialty  suites  (see  Assumptions  la  and  lb  above).  These  associations 
were  accomplished  by  identifying  the  Avionics  tasks  requiring  troubleshooting  (when  the 
technician  used  aircraft  controls  eind  displays  to  determine  operational  condition).  In  such  tasks, 
technicians  must  interpret  system  operating  characteristics  to  isolate  malfunctions.  The  systems 
and  subsystems  identified  as  relevant  included:  attack  control,  radar,  infrared,  laser,  instruments, 
displays,  flight  control,  communication,  navigation,  satellite  communications,  identification 
(defensive  and  offensive),  and  defensive  or  offensive  electronic  warfare  systems. 

Similarly,  E  &  E  on-equipment  systems  include  direct  and  alternating  current,  gas  turbine 
compressors  and  auxiliary  power  units,  landing  gear,  anti-skid,  and  nose  wheel  steering, 
electronic  engine  control,  ignition,  and  starting.  Other  E  &  E  systems  include  lighting,  master 
caution  and  warning,  take-off  warning,  flight  controls,  cargo  door  and  cargo  delivery  equipment, 
non-electro  static  application  windows,  anti-icing,  fire  and  overheat  warning,  fire  suppression, 
fuel  control,  liquid  cooling,  air  conditioning,  bleed  air,  cabin  pressurization,  and  auxiliary 
pressurization,  oxygen,  and  aircraft  utility  systems. 

Table  A-1  shows  an  example  of  the  system  selection  strategy  for  the  F-16  Block  40/42 
aircraft.  The  first  four  columns  (Block;  WUC;  Occurrences;  Description)  contain  actual  AFDS 
data.  The  last  two  columns  (E  &  E;  Avionics)  indicate  the  number  of  applicable  instances  by 
system. 
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Table  A-1  Example  of  Filtering  by  WUC  and  Aircraft  System/Subsystem 


Block 

WUC 

Occurrences 

Description 

E&E 

Avionics 

40 

46E 

339 

FUEL  INDICATING-CON 

339 

42 

46E 

118 

FUEL  INDICATING-CON 

118 

40 

46F 

803 

FUEL  TANKS  EXTERNAL 

42 

46F 

401 

FUEL  TANKS  EXTERNAL 

40 

47A 

922 

LIQ  OXY  SYS  SUP&DST 

922 

42 

47A 

1094 

LIQ  OXY  SYS  SUP&DST 

1094 

40 

49A 

107 

FIRE  DETECTION  SYS 

107 

42 

49A 

48 

FIRE  DETECTION  SYS 

48 

40 

49B 

10 

OVERHEAT  DETCTN  SYS 

10 

42 

49B 

5 

OVERHEAT  DETCTN  SYS 

5 

40 

51A 

86 

PRIMARY  FLIGHT  INST 

86 

42 

51A 

71 

PRIMARY  FLIGHT  INST 

71 

40 

51B 

26 

ARTIFICAL  REF  INSTR 

26 

42 

51B 

24 

ARTIFICAL  REF  INSTR 

24 

40 

51C 

20 

SECONDARY  INSTRUMENTS 

20 

42 

51C 

22 

SECONDARY  INSTRUMENTS 

22 

40 

51D 

4 

STANDBY  INSTRUMENTS 

4 

Using  these  figures,  the  team  calculated  a  one-year  total  of  103,541  relevant  maintenance 
actions.  The  next  step  was  to  calculate  a  representative  12-month  figure  for  the  total  number  of 
Avionics  and  E  &  E  maintenance  tasks  that  involve  troubleshooting.  The  resulting  calculations 
were  16,742  for  Avionics  and  37,747  for  E  &  E.  Approximately  50%  of  the  initial  data  were 
determined  to  involve  Avionics  and  E  &  E  troubleshooting  according  to  these  selection  criteria. 

Following  these  calculations,  the  Task  Time  delta  observed  between  the  To-be  and  As-is 
test  conditions  for  the  system  specific  expert  and  novice  user  groups  was  multiplied  by  the 
relevant  number  of  repairs  involving  troubleshooting,  for  both  Avionics  and  E  &  E  systems.  The 
result  was  a  calculated  savings  of  45  minutes  per  Avionics  task  and  55  minutes  per  E  &  E  task 
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(see  Table  A-2  below).  Multiplying  the  quantity  of  relevant  repairs  by  the  average  time  savings 
revealed  a  fleet-wide  savings  of  12,557  Avionics  troubleshooting  hours,  and  34,601  E  &  E 
troubleshooting  hours.  Table  A-2  illustrates  the  calculations  of  an  anticipated  total  savings  of 
47,158  hours  for  a  12-month  period.  Dividing  47,158  hours  by  the  clock  hours  available  in  a 
year  (365*24  =  8760)  reveals  AMIT  potential  to  add  the  equivalent  of  5.4  aircraft  to  the 
available  fleet  (47,158  /  8760  =  5.4). 

Table  A-2  F-16  Block  40/42  Aircraft  Anticipated  Savings 


12  months  of  Write-ups 
Total  E  &  E  FC 

A/C  Savings 

E  &  E  Time  Savings 
Session  Total 

(Min)  Total  (Min)  (Hrs) 

Avionics  Time  Savings 

Session  Total 

(Min)  Total  (Min)  (Hrs) 

Combined 

Totals 

(Hrs) 

103541  37747  16742 

55  2076096  34601 

45  753408  12557 

47158 

The  calculated  time  savings  used  to  calculate  man-hour  cost  savings.  Assuming  an 
average  crew  size  of  two  (some  tasks  take  more  than  two;  some  only  accommodate  one),  the 
increase  in  aircraft  availability  hours  (47,158)  is  multiplied  by  two  to  provide  the  anticipated 
total  man  hour  savings  across  all  relevant  maintenance  actions  (94,316).  This  time  can  then  be 
converted  to  a  dollar  figure  by  multiplying  the  average  labor-hour  cost.  Table  A-3  (extracted 


from  API  65-503,  Table  A20-1)  displays  the  cost  per  hour  of  the  three  most  likely 


troubleshooting  grades.  The  average  of  the  three  is  $29.99. 


Tabic  A-3  Pay  Rates  for  Typical  Maintenance  Technician  Ranks 


Rank 

Annual 

Month 

Week 

Day 

Hour 

E-6 

$71,906 

$5,992.16 

$1,382.81 

$276.56 

$34.57 

E-5 

$63,026 

$5,252.13 

$1,212.03 

$242.41 

$30.30 

E-4 

$52,200 

$4,349.97 

$1,003.84 

$200.77 

$25.10 
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Using  the  above  methodology,  the  calculated  one-year  dollar  savings  for  F-16  Block 


40/42  fleet  would  be  $2,828,561 .  See  Table  A-4  for  calculations. 


Table  A-4  Anticipated  Manpower  Savings  for  Block  40/42 


Combined 

Totals  (Hrs) 

Manpower  Savings 

Crew  Total  $  per 

size  (Fleet)  hour  Cost 

47158 

2  94316  $29.99  $2,828,561 

A.3.  Extrapolation  beyond  Block  40/42 

As  Table  A-1  illustrates,  extrapolation  beyond  the  test  bed  aircraft  (F-16C  Block  40/42) 
is  possible,  but  carries  an  inherent  degree  of  reduced  confidence.  Certainly,  the  total  dollar 
savings  will  increase  as  additional  weapons  systems  are  factored  into  the  analysis.  These 
increases,  however,  are  mitigated  by  the  fact  that  similar  troubleshooting  techniques  or  “rules-of- 
thumb,”  while  applicable  to  the  F-16C  Block  40/42  and  other  F-16  blocks,  may  not  apply  as 
directly  to  other  airframes. 

Extrapolation  to  the  entire  F-16  fleet,  consisting  of  1,314  aircraft,  generates  a  135,199 
hours  of  aircraft  availability  (Mission  Capability)  across  Avionics  and  E  &  E  tasks  see  Table  A- 
5).  This  equates  to  15  additional  aircraft  in  the  fleet.  To  compute  the  total  man  hour  savings 
multiply  by  two  based  on  average  maintainer  crew  size  would  produce  an  anticipated  savings  of 
270,398  man  hours  at  a  cost  savings  of  $8,109,224  annually  (see  Table  A-6).  See  Table  A-5  and 
A-6  for  computations. 
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Table  A-5  F-16  Fleet  Calculations  -  Aircraft  Hours 


A/C 

12  months  of  Write 

-ups 

A/C  Savings 

Type 

# 

Total 

E&  E 

FC 

E  &  E  Time  Savings  (Min) 

FC  Time  Savings  (Min) 

Session 

Total 

Total 

Session 

Total 

Total 

(Min) 

(Min) 

(Hrs) 

(Min) 

(Min) 

(Hrs) 

15 

62 

8638 

3312 

859 

55 

182160 

3036 

45 

38664 

644 

25 

202 

46428 

17210 

5208 

55 

946572 

15776 

45 

234360 

3906 

30/32 

396 

74719 

23321 

11484 

55 

1282644 

21377 

45 

516780 

8613 

40/42 

402 

103541 

37747 

16742 

55 

2076096 

34602 

45 

753408 

12557 

50/52 

252 

77266 

25982 

14494 

55 

1429032 

23817 

45 

652212 

10870 

F-16 

Total 

1314 

310591 

107573 

48787 

5916504 

98608 

2195424 

36590 

Table  A-6  F-16  Fleet  Calculations  -  Manpower 


F-16 

Manpower  Savings 

Crew 

Total 

$  per 

Block 

#  of  A/C 

Combined 

Totals 

Equivalent 

Additional 

size 

(Fleet) 

hour 

Cost 

(Hrs) 

A/C 

15 

62 

3680 

0.42 

2 

7361 

$29.99 

$220,750 

25 

202 

19682 

2,25 

2 

39364 

$29.99 

$1,180,538 

30/32 

396 

29990 

3.42 

2 

59981 

$29.99 

$1,798,824 

40/42 

402 

47158 

5.38 

2 

94317 

$29.99 

$2,828,561 

50/52 

252 

34687 

3.96 

2 

69375 

$29.99 

$2,080,550 

F-16 

Totals 

1314 

135199 

15.43 

2 

270398 

$29.99 

$8,109,224 

The  same  search  and  selection  criteria  can  be  used  to  calculate  potential  savings  for  other 
Weapon  Systems  (WS)  such  as  the  A-10,  F-15,  and  C-130.  While  the  confidence  level  in  such 
extrapolations  is  less  than  with  that  for  the  F-16s,  an  equally  noteworthy  degree  of  savings  can 
be  projected.  Tables  A-7  and  A-8  detail  the  extrapolation  of  savings  to  the  airframes  mentioned 
above,  using  actual  historical  data  in  terms  of  maintenance  actions  taken. 
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Table  A-7  A-10,  F-15,  and  C-130  Fleet  Calculations 


A/C 

12  months  of  Write-ups 

A/C  Savings 

Type 

# 

Total 

E&E 

FC 

E&E  Time  Savings  (Min) 

FC  Time  Savings  (Min) 

Session 

Total 

Total 

Session 

Total 

Total 

(Min) 

(Min) 

(Hrs) 

(Min) 

(Min) 

(Hrs) 

A-10A 

246 

50933 

22442 

9031 

55 

1234332 

20572 

45 

406404 

6773 

104 

26112 

11263 

2726 

55 

619476 

10325 

45 

122688 

2045 

392 

73440 

36499 

9110 

55 

2007456 

33458 

45 

409968 

6833 

224 

51247 

23071 

7445 

55 

1268916 

21149 

45 

335016 

5584 

F-15 

Totals 

720 

150799 

70833 

19281 

3895848 

64932 

867672 

14462 

AC130H 

8 

8263 

3046 

1006 

55 

167508 

2792 

45 

45252 

754 

AC130U 

15 

8081 

3094 

871 

55 

170148 

2836 

45 

39204 

653 

C130E 

167 

110448 

56206 

11678 

55 

3091308 

51522 

45 

525528 

8759 

C130H 

278 

175140 

76212 

30638 

55 

4191660 

69861 

45 

1378728 

22979 

C-130J 

34 

12487 

4762 

3046 

55 

261888 

4365 

45 

137052 

2284 

EC130H 

10 

4915 

1936.8 

1366 

55 

106524 

1775 

45 

61452 

1024 

HC130N 

g 

4754 

2057 

607 

55 

113124 

1885 

45 

27324 

455 

HC130P 

22 

12746 

5777 

2143 

55 

317724 

5295 

45 

96444 

1607 

LC130H 

10 

5467 

2155 

710 

55 

118536 

1976 

45 

31968 

533 

MC130E 

14 

20808 

7493 

3936 

55 

412104 

6868 

45 

177120 

2952 

MC130H 

20 

17923 

6809 

3317 

55 

374484 

6241 

45 

149256 

2488 

MC130P 

27 

26866 

11467 

3646 

55 

630696 

10512 

45 

164052 

2734 
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Table  A-8  A-10,  F-15,  and  C-130  Fleet  Calculations 


Manpower  Savings 

Crew 

Total 

$  per 

Type 

#  of  A/C 

Combined 

Totals 

Equivalent 

size 

(Fleet) 

hour 

Cost 

(Mrs) 

Additional  A/C 

A-10A 

246 

27346 

3.12 

2 

54691 

$29.99 

$1,640,189 

104 

12369 

1.41 

2 

24739 

$29.99 

$741,917 

392 

40290 

4.6 

2 

80581 

$29.99 

$2,416,618 

224 

26732 

3.05 

2 

53464 

$29.99 

$1,603,397 

F-15 

Totals 

720 

79391 

9,06 

CN 

158784 

$29.99 

$4,761,932 

AC130H 

8 

3546 

0.4 

2 

7092 

$29.99 

$212,689 

AC130U 

15 

3489 

0.4 

2 

6978 

$29.99 

$209,282 

C130E 

167 

60281 

6.88 

2 

120561 

$29.99 

$3,615,630 

C130H 

278 

92840 

10.6 

2 

185680 

$29.99 

$5,568,531 

C-130J 

34 

6649 

0.76 

2 

13298 

$29.99 

$398,807 

EC130E 

3 

EC130H 

10 

2800 

0.32 

2 

5599 

$29.99 

$167,920 

HC130N 

9 

2341 

0.27 

2 

4682 

$29.99 

$140,401 

HC130P 

22 

6903 

0.79 

2 

13806 

$29.99 

$414,030 

LC130H 

10 

2508 

0.29 

2 

5017 

$29.99 

$150,454 

MC130E 

14 

9820 

1.12 

2 

19641 

$29.99 

$589,028 

MC130H 

20 

8729 

1 

2 

17458 

$29.99 

$523,565 

MC130P 

27 

13246 

1.51 

2 

26492 

$29.99 

$794,483 

Note  the  anticipated  savings  for  the  F-15  fleet,  which  equates  to  nearly  80,000  hours  of 
aircraft  availability.  This,  in  turn,  represents  an  additional  nine  F-15  aircraft  in  the  fleet.  The  C- 
130  savings  were  not  summed  due  to  the  diverse  missions  associated  with  each  model. 
However,  examination  of  those  C-130  aircraft  with  similar  missions  (specifically  the  -E,  -H,  and 
-J),  the  combined  aircraft  availability  savings  of  159,770  hours  equates  to  an  additional  18 
aircraft.  Combined  man  hour  savings  and  cost  for  these  three  models  are  3 19,540  man  hours  and 
$9,582,968. 
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Appendix  B:  Transition  Activities  and  Recommendations 

B.1.  Program  Introduction 

Aircraft  Maintenance  Intuitive  Troubleshooting  (AMIT)  was  conceived  at  an 
AFRL/MAJCOM  technical  interchange  meeting,  with  the  realization  that  technician  performance 
was  an  under  explored  and  possibly  significant  contributor  to  aircraft  availability  and  readiness. 
Heretofore,  the  Air  Force  (AF)  has  given  much  attention  to  maintenance  levels,  repair  locations, 
spares,  tools  and  test  equipment,  technician  training,  specialty  codes,  and  even  to  job 
performance  rating.  Little  attention,  however,  has  been  paid  to  factors  affecting  the  technician’s 
performance  -  the  “how”  and  “why”  technicians  perform  the  way  they  do  on  the  flightline.  How 
do  technicians  successfully  return  highly  complex  aircraft  to  service  in  spite  of  the  known 
limitations  and  challenges  imposed  by  the  maintenance  process  and  related  support  and  supply 
systems? 

Given  this  line  of  thinking,  the  Logistics  Readiness  Branch  of  the  Human  Effectiveness 
Directorate  of  the  Air  Force  Research  Laboratory  (AFRL/HEAL)  initiated  a  6.3  advanced 
research  and  development  (R&D)  program  and  awarded  the  AMIT  contract  to  NCI  Information 
Systems,  Incorporated  (NCI)  in  December  2003.  NCI  is  an  information  technology  services 
provider  with  relevant  skills  in  the  aircraft  maintenance  and  maintenance  management  domains. 

B.2.  Transition  Overview 

AFRL/HEAL  placed  highest  priority  and  value  on  transition  at  the  project  outset,  which 
continued  throughout  the  program.  The  strategic  objectives  of  AMIT  transition  were  simple: 

•  Coordinate  with  the  senior  AF  leadership  to  ensure  that  AMIT  fits  into  the  big  picture  of 

logistics  transformation  initiatives. 
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•  Find  the  right  transition  agent  to  take  over  the  implementation  role  post-research, 
development,  and  testing 

•  Coordinate  a  deliberate  Technology  Transition  Plan  (TTP)  with  the  transition  agent  and 
the  operational  users  that  would  be  included  in  the  implementation  effort 

•  Coordinate  with  operational  users  to  plan  ahead  for  future  POM  wedges  and  funding  for 
an  AMIT  pilot  implementation 

These  objectives  were  all  critical  and  are  necessarily  prioritized. 

The  AMIT  Team  conducted  the  research  and  developed  the  capability  within  the 
guidelines  and  in  the  spirit  of  critical  AF  programs  such  as  the  Expeditionary  Logistics  for  the 
21st  Century  (eLOG21),  the  Expeditionary  Combat  Support  System  (ECSS),  and  the  Global 
Combat  Support  System  -  Air  Force  (GCSS-AF). 

B.3.  Approach 

The  AMIT  transition  approach  hinged  on  a  philosophy  of  discovering  and  frequently 
engaging  potential  transition  agents,  operational  sponsors,  and  funding  sources.  For  the  purposes 
of  transition,  there  was  no  single  preferred  customer.  The  primary  rationale  for  this  approach 
was  AMIT,  as  a  concept,  was  generically  applicable  to  all  flightline  maintenance  troubleshooting 
activities,  regardless  of  Department  of  Defense,  Air  Force,  MAJCOM,  Base,  Group,  or  Squadron 
mission.  The  underlying  transition  theme  was  to  support  Combatant  Commanders  fighting  the 
global  war  on  terrorism  (GWOT)  and  conducting  other  combat  operations. 

In  approaching  transition  throughout  the  research,  development,  and  testing  phases  of  the 
program,  the  AMIT  Team  put  considerable  thought  and  effort  into  finding  where  the  AF  is 
heading  with  its  Information  Technology  (IT)  transformation  vision.  We  learned  how  senior 
policy  makers  at  Air  Staff  plan  to  address  enterprise  level  initiatives  -  programs  like  eLog21, 
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ECSS,  GCSS-AF,  and  Air  Force  Smart  Operations  for  the  Century  (AFSO  21).  Awareness 
of  challenging  initiatives  impacting  the  logistics  domain  in  both  the  near  and  long  term  were  also 
important  -  force  sizing  and  enterprise  transformation  actions  like  Task  Force  720  (TF  720)  and 
Repair  Enterprise  21  (RE  21)  -  when  addressing  the  future  operational  flightline  environment 
that  included  the  AMIT  concept. 

B.4.  Transition  “Roadshow”  Team 

In  order  to  maximize  transition  dollars,  the  AMIT  PM  formed  the  AMIT  “Roadshow 
Team”  very  early  in  the  program.  The  basic  charter  of  the  Team  was  to  take  the  AMIT  message 
to  logistics  leaders  at  all  levels  of  the  Air  Force,  joint  service  commands,  and  mid-tier 
maintenance  managers  at  MAJCOM,  Group,  and  Squadron  levels. 

The  Team  was  comprised  of  core  AMIT  personnel,  including  the  AFRL/HEAL  Program 
Manager  (PM),  Mr.  Chris  Curtis,  two  seasoned  HEAL  aircraft  maintenance  officers,  Captains 
Vaughan  Whited  and  Derrick  Barthol,  and  the  NCI  PM/transition  lead,  Mr.  Charles  Botello.  The 
Roadshow  Team  succeeded  in  presenting  AMIT  to  the  logistics  warfighter  on  many  fronts  -  in 
the  offices  of  the  senior  logistics  policy  staffs  at  the  Pentagon,  the  technical  training 
schoolhouses,  the  MAJCOM  headquarters,  operational  aircraft  maintenance  units,  and  in  the 
aircraft  depot  maintenance  hangars  and  baek  shops.  Figure  B-1  summarizes  milestone  contacts 
with  potential  transition  agents,  operational  sponsors,  and  potential  funding  agents.  The  fishbone 
chart  chronicles  actual  transition  activities  and  accomplishments  during  the  course  of  the  AMIT 
program  -  May  2004  through  contract  end  in  January  2007. 
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AMIT  TRANSITION  STRATEGY  TIMELINE... 

...Some  key  Highlights 
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AF  AMIT  Transition  Strategy 
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AF/A4M‘'  =  Force  Dev  Cell 


Figure  B-1.  AMIT  Transition  Activity  Timeline 


The  transition  team  met  weekly  for  the  duration  of  the  program.  In  the  first  6  months  of 
the  program,  the  team  had  identified  many  of  the  critical  players  in  the  AF  infrastructure,  and 
placed  a  priority  on  selecting  viable  transition  paths  and  strategies.  By  May  of  2004,  the  AMIT 
transition  strategy  was  initially  developed,  and  the  team  planned  to  establish  ongoing  dialog  with 
key  organizations  representing  the  operational  warfighter  and  our  primary  AMIT  target  -  the 
aircraft  maintenance  flightline  technician. 

The  AMIT  transition  strategy  was  outlined  in  an  informal  TTP.  The  TTP  helped  our 
transition  team  lay  out  the  focus  of  the  AMIT  message,  identify  and  list  the  intended  target 
organizations  and  key  personnel,  capture  the  strategic  roadmap  on  how  to  coordinate  planned 
visits,  and  measure  our  level  of  progress  in  our  Roadshow  team  efforts.  With  the  TTP  as  the 
guiding  document,  the  team  continually  adjusted  and  re-baselined  execution  of  planned  transition 
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activities.  On  a  v^eekly  basis,  the  team  reviewed  strategic  objectives  and  executed  activities  to 
meet  those  goals  in  30,  60  and  90-day  intervals.  After  key  visits,  results  were  documented  in  trip 
reports.  The  critical  data  captured  from  these  trips  included  an  analysis  of  whether  objectives 
and  takeaways  were  achieved,  notes  of  key  discussions  and  resultant  action  items  that  aided  in 
transforming  the  transition  strategy  and  in  updating  the  TTP,  accordingly. 

One  of  the  most  intriguing  transition  obstacles  the  AMIT  Roadshow  team  consistently 
had  to  overcome  was  the  operational  users’  lack  of  imderstanding  regarding  what  AFRL/HEAL 
was  trying  to  do  with  performance  enhancement  initiatives  like  AMIT.  Regularly  visiting  the 
major  commands,  operational  bases,  joint  service  commands.  Headquarters  Air  Force,  and 
selected  logistics-related  conferences  proved  to  be  beneficial. 

One  of  the  primary  drivers  in  the  AMIT  transition  strategy  was  a  comprehensive, 
working  knowledge  of  where  AMIT  could  fit  into  an  operational  environment  and  how  AMIT 
could  be  sustained  within  the  information  technology  upgrades  envisioned  within  the  next  5-10 
years.  As  we  envisioned  in  the  transition  strategy  in  years  one  and  two  of  the  program,  an  AMIT 
pilot  application  or  limited-scale  implementation  became  the  focus  of  transition  discussions,  as 
AMIT  was  developed  to  integrate  into  the  GCSS-AF  Integration  Framework.  In  the 
implementation  concept,  the  AMIT  service  would  be  accessible  through  the  AF  Portal,  where  the 
operational  user  could  gain  access  through  the  single  log-in  point  of  entry.  In  this  IT-secure 
environment,  the  technician  would  have  at  his  or  her  disposal  interfaees  to  aircrew  debriefing 
documentation,  the  relevant  aircraft  technical  orders,  the  historical  data  for  aircraft 
systems/subsystems/components,  shop  level  logbook  data,  and  the  means  to  collaborate  with 
others. 
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B.5.  Results,  Discussion,  and  Recommendations 

The  ACC  logistics  staff  has  fully  embraced  and  formally  adopted  the  AMIT  concept  in 
submitting  a  Programmed  Objective  Memorandum  (POM)  line  item  wedge  for  AMIT  in 
addressing  its  Fiscal  Year  2008  combat  logistics  support  needs.  In  the  August-September  2006 
timeframe,  when  racked  and  stacked  at  ACC  amongst  other  logistics  operational  CAF  needs  for 
the  F-16  Block  40/50,  AMIT  ranked  13*  of  approximately  24  inputs  on  the  list. 

Since  Brig  Gen  David  Gillette  (then  AF/A4M,  and  now  ACC/A4)  signed  the  AMIT 
Memorandum  of  Understanding  in  late  2005,  there  has  been  significant  personnel  turnover  at 
both  AF/A4M  and  AF/A4MM,  the  chief  policy  makers  for  all  Air  Force  maintenance  and  the 
maintenance  information  systems  portfolio  managers  within  the  eLOG21  Portfolio  Management 
Program  of  AF/A4I. 

In  December  2005,  Brig  Gen  Gillette  gave  our  AMIT  Roadshow  team  direction  to  pursue 
an  implementation  of  AMIT  within  the  principles  of  the  ECSS,  rather  than  the  pursuit  of  an 
ACC-sponsored  Advanced  Concept  Technology  Demonstration  (ACTD).  His  preference  was  to 
focus  near-term  AMIT  transition  efforts  on  a  pilot  implementation  vice  using  critical  resources 
for  yet  another  demonstrator. 

Therefore,  the  AMIT  team  has  embarked  on  integrating  AMIT  as  a  pilot  into  pre-existing 
programs  in  lieu  of  doing  additional  AMIT  user  assessments.  The  programs  Brig  Gen  Gillette 
suggested  included  GCSS-AF,  ECSS,  and  Point  of  Maintenance  (POMx).  Collaboration  work  is 
being  performed  with  each  of  these  efforts.  However  GCSS-AF  is  only  the  technical  framework 
that  will  provide  access  to  the  AMIT  application.  ECSS  just  concluded  its  source  selection 
process  for  the  System  Integrator  and  is  at  least  3-5  years  from  hosting  the  AMIT  capability. 
POMx  is  anticipated  to  be  absorbed  into  the  higher  level  requirement  of  IMDS. 
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The  primary  objectives  of  an  AMIT  deployment  plan  are  to  help  the  Air  Staff,  as  the 
champion  of  the  AMIT  concept,  and  the  operational  major  command  to  form  a  basic,  deliberate, 
and  incremental  approach  to  instituting  AMIT  into  base  level  operations.  This  deployment  plan 
will  require  full  coordination  with,  and  direct  support  from,  the  owning  major  command. 

Consideration  of  the  AMIT  pilot  implementation  options  is  a  function  of  full  and 
thorough  coordination  with  the  major  command  that  is  stepping  up  to  take  ownership  of  the 
transition  agent  role,  such  as  the  ACC  as  the  primary  CAP  lead.  The  common  denominator  with 
these  options  is  that  the  technical  data  can  be  provided  by  the  respective  technical  order 
distribution  accounts  (TOD As)  in  the  aircraft  sustainment  groups,  as  well  as  the  Original 
Equipment  Manufactures  (OEMs).  The  historical  data  can  be  accessed  through  the  Air  Force 
Portal  from  AFDS  and  the  respective  weapons  system’s  data  in  IMDS.  Logbook  data  can  be 
gathered  from  the  base  level  organization  sponsoring  the  pilot  application.  Based  on  our 
transition  activities  over  the  past  1 8  months,  the  AMIT  team  recommends  the  options  described 
in  the  paragraphs  below. 

•  CAF  F- 16  Fixed  Wing  Aircraft 

The  F-16  aircraft  is  the  top  operational  contender  for  an  AMIT  pilot  implementation.  It 
has  been  our  platform  of  choice  since  the  development  program  inception  in  late  2003.  We  have 
conducted  our  field  test  using  F-16  Block  42  aircraft  assigned  to  the  56th  Maintenanee  Group  at 
Luke  AFB.  Both  ACC  and  AETC,  with  its  19th  AF  logistics  and  operations  contingent,  have 
fully  supported  the  FDT  at  Luke  AFB.  The  508th  Aircraft  Sustainment  Group  at  Hill  AFB,  Utah 
has  been  tremendously  supportive  and  provided  the  F-16  technical  data  in  Standard  Generalized 
Markup  Language  for  our  AMIT  team  to  use. 
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Depending  on  availability  of  aircraft  and  other  critical  resources,  as  well  as  non¬ 
interference  operational  considerations,  an  AMIT  pilot  application  could  be  implemented  at  one 
of  the  following  bases:  Luke  AFB,  Hill  AFB,  or  Shaw  AFB.  In  addition,  implementation  at 
Burlington  ANG,  Vermont  could  be  accomplished  within  the  operating  guidelines  of  the 
ANG/ACC  CAF  Total  Force  Program. 

•  AFSOC  MQ-1  Predator  Unmanned  Air  Vehicle 

The  MQ-1  Predator  is  another  viable  platform,  soon  to  be  within  the  operational 
inventory  of  AFSOC  mission  commitments.  The  senior  leadership  in  the  AFSOC  logistics 
(A4M)  community  is  highly  supportive  of  the  AMIT  concept  and  wants  to  consider  an  AFSOC 
pilot  implementation.  The  AMIT  team  understands  that  the  technical  data  for  the  MQ-1  can  be 
provided  in  Interactive  Electronic  Technical  Manual  (lETM)  format. 

•  CAF  H-60  Rotary  Wing  Aircraft 

The  H-60  helicopter  is  also  a  viable  candidate  platform  for  an  AMIT  pilot  application. 
H-60  helicopter  maintenance  troubleshooting,  data  interfaces  and  other  maintenance  processes 
essentially  mirror  the  fixed  wing  assets. 

B.6.  Transition  Summary 

The  AMIT  transition  team  followed  the  right  path  in  its  transition  strategy,  and  started 
that  path  early.  Operational  users  at  all  levels  were  convinced  that  AMIT  was  the  right 
teehnology  with  high  potential  to  be  a  force  multiplier  for  one  of  our  most  critical  resources  -  the 
flightline  maintenance  technician.  General  officers,  Senior  Executive  Service  civilians,  and 
Expert/Novice  technicians  alike  embraced  the  AMIT  concept.  The  universal  thread  of  their 
comments  focused  on  the  need  to  have  an  AMIT  capability  operational  as  soon  as  possible. 
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The  strategy  used  by  the  AMIT  team  proved  to  be  beneficial  in  successfully  transitioning 
the  AMIT  concept  to  key  organizations  that  bear  direct  responsibilities  in  transforming  and 
sustaining  the  resources  for  the  operational  logistics  forces  involved  in  the  GWOT  efforts  and 
other  war  fighting  activities  worldwide. 
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